Debian Codegraph @

doc/LARGEFILE

raw
1 The state from 1.32.0 on:
2 =========================
3
4 Yet again, things had to change. The previous state works fine for Unix (adjacent)
5 sytems, also MinGW stuff on Windows, but it still suffers from off_t being shifty
6 and not behaving as I expect, even, when people try to build mpg123 with rogue
7 compilers like MSVC, where off_t is always 32 bits, no largefile support, the whole
8 idea falls flat. I was pushed to introduce API without off_t where the user can
9 provide I/O and ensure largefile support and whatnot without being encumbered by
10 offsets possibly limited to 32 bits.
11
12 So this is a new set of API functions with the explicit suffix 64 (not _64), notably
13 simple functions like
14
15 int64_t mpg123_tell64()
16
17 but also
18
19 int mpg123_reader64()
20
21 which takes callbacks that work on int64_t, always. The internal reader code path
22 in libmpg123 uses either 32 bit or 64 bit offsets, but throughout the library,
23 it is 64 bits being used as if we supported large files everywhere. And in fact,
24 we do, if the user provides callbacks that are smart enough.
25
26 All existing offset-sensitive code paths are to be rephrased in terms of the new
27 portable API (portable, because the platform specifics of off_t implementation are
28 not present). All old API entry points in that regard become wrappers over this
29 portable API.
30
31 The build of libmpg123 needs explicit awareness of largefile support to properly
32 reason about which wrapper uses which types. It makes use of off_t at default
33 size and explicit off64_t if available. This results in this for a typical 32 bit
34 Linux system:
35
36 int64_t mpg123_tell64(); // passing on the internal int64_t value
37 off_t mpg123_tell(); // converting to 32 bit off_t (overflow check)
38 off_t mpg123_tell_32(); // converting to 32 bit off_t (overflow check)
39 off64_t mpg123_tell_64(); // converting to 64 bit off_t (no overflow)
40
41 A 64 bit Linux would have only these:
42
43 int64_t mpg123_tell64(); // passing on the internal int64_t value
44 off_t mpg123_tell(); // converting to 64 bit off_t (no overflow)
45 off_t mpg123_tell_64(); // converting to 64 bit off_t (no overflow)
46
47 This would look the same on a 32 bit system that also went for fixed 64 bit offsets,
48 or if you
49
50 #define _FILE_OFFSET_BITS 64
51
52 during build on a sensitive system. On a funny system that has 64 bit native integers
53 but still 32 bit off_t, this would also be:
54
55 int64_t mpg123_tell64(); // passing on the internal int64_t value
56 off_t mpg123_tell(); // converting to 32 bit off_t (overflow check)
57 off_t mpg123_tell_32(); // converting to 32 bit off_t (overflow check)
58
59 If this weird system would be also largefile-sensitive, you get the additional
60
61 off64_t mpg123_tell_64(); // converting to 64 bit off_t (no overflow)
62
63 entry point.
64
65 The whole point here: If you stick to mpg123_tell64(), you always can assume
66 64 bit offsets on any platform. If you also provide your own reader functions, you
67 can ensure it. Even if we don't properly do large file I/O in libmpg123 itself in
68 an MSVC build, users can provide it with their application. Providing a full mpg123
69 program binary along with it in that environment is not a current target. You want
70 the library.
71
72 The point that every platform gets the _64 suffix means that the header can still
73 do the renaming of function calls if client code defines _FILE_OFFSET_BITS and not
74 depend on any build-time switches for that. This is what we do officially now with
75 headers that are not preprocessed anymore, independent of platform.
76
77 The build, just assuming int64 is available, needs to know some bits:
78
79 - SIZEOF_OFF_T==8: Is the system 64 bit clean, anyway, with 64 bit off_t?
80 - LFS_LARGEFILE_64: Is there explicit 64 bit I/O for the 32 bit system?
81 This includes availability of off64_t.
82
83 If SIZEOF_OFF_T==8, there will be all 64 bit internal code and two wrappers:
84
85 off_t mpg123_tell(); // converting to 64 bit off_t (no overflow)
86 off_t mpg123_tell_64(); // converting to 64 bit off_t (no overflow)
87
88 The second question isn't even needed. If SIZEOF_OFF_T==4, you get
89
90 off_t mpg123_tell(); // converting to 32 bit off_t (overflow check)
91 off_t mpg123_tell_32(); // converting to 32 bit off_t (overflow check)
92
93 and the additional decision of LFS_LARGEFILE_64 gives actual 64 bit internal
94 reader code and the wrapper
95
96 off64_t mpg123_tell_64(); // converting to 64 bit off_t (no overflow)
97
98 along with that. There is nothing more to know about a system. I do not care how
99 large your long is. I only ask
100
101 1. How big is your off_t?
102 2. If small, can you give me some off64_t?
103
104 That's all what there is to it. The build is not even interested in whether off_t
105 changes size. This is a detail on the client side that it then gets the off64_t
106 versions. I do not support a system that allows changing off_t size _without_
107 off64_t being available.
108
109
110 The state from 1.15.4 on:
111 =========================
112
113 Regarding largefile setup, client apps can be built three ways:
114
115 1. _FILE_OFFSET_BITS == 64 (header maps to mpg123_open_64)
116 2. _FILE_OFFSET_BITS == 32 (header maps to mpg123_open_32)
117 3. _FILE_OFFSET_BITS == <nothing> (header maps to mpg123_open)
118
119 The libmpg123 build needs to be prepared for everything. Also, it needs to keep
120 in mind the days before introducing large file support --- binaries should still
121 work with updated libmpg123. So, mpg123_open should always match what is the
122 default build on a platform without added settings. Those are the platform
123 variants:
124
125 1. 64 bit native system, long == off_t
126 libmpg123: mpg123_open
127 lfs_alias: mpg123_open_64 -> mpg123_open
128 lfs_wrap: <none>
129
130 2. largefile-sensitive, long = 32, off_t = 64 (if enabled)
131 libmpg123: mpg123_open_64
132 lfs_alias: mpg123_open_32 -> mpg123_open
133 lfs_wrap: mpg123_open -> mpg123_open_64
134
135 3. largefile, long = 32, off_t = 64 (FreeBSD)
136 libmpg123: mpg123_open
137 lfs_alias: mpg123_open_64 -> mpg123_open
138 lfs_wrap: <none>
139
140 This is what mpg123 does in version 1.15.4 and it works. Well, for cases 1
141 (Linux/Solaris x86-64) and 2 (Linux/Solaris x86). Case 3 needs to be added
142 properly. Actually, let's have a second look at case 2: When mpg123 is built
143 with --disable-largefile:
144
145 2a. largefile-sensitive, mpg123 built with off_t = 32 == long
146 libmpg123: mpg123_open
147 lfs_alias: mpg123_open_32 -> mpg123_open
148 lfs_wrap: <none>
149
150 So, this is still correct. Now, what about case 3? What does mpg123 do
151 currently, as of 1.15.4?
152
153 3a. largefile, long = 32, off_t = 64 (... and mpg123 not really aware of that)
154 libmpg123: mpg123_open
155 lfs_alias: mpg123_open_32(long) -> mpg123_open(off_t)
156 lfs_wrap: <none>
157
158 This is _wrong_. Luckily, this does not cause binary compatibility issues, as
159 mpg123_open_32 won't be called by anyone unless that someone tries to define
160 _FILE_OFFSET_BITS=32, which is nonsense. Perhaps old FreeBSD binaries before
161 LFS times? Well, back then, there was no libmpg123. So let's ignore that case.
162 The issue at hand is that the alias should be from mpg123_open_64 to
163 mpg123_open, for clients that insist on defining _FILE_OFFSET_BITS=64.
164
165 The change needed now is to fix the naming and also change the type the
166 alias functions use: It is not long int anymore!
167
168 Let's revisit case 1 for a moment: My old lfs_alias.c provides for the case
169 lfs_alias: mpg123_open -> mpg123_open_64. Is that actually possible?
170 What means enforcing _FILE_OFFSET_BITS=64 from the outside, which _could_
171 happen when libmpg123 is included someplace and folks are on the wrong side
172 of paranoid regarding this. So, there is
173
174 1a. 64 bit native system, long == off_t = 64 and _FILE_OFFSET_BITS=64
175 libmpg123: mpg123_open_64
176 lfs_alias: mpg123_open -> mpg123_open_64
177 lfs_wrap: <none>
178
179 (Works also for any system with long == off_t in any width)
180 Likewise, there is largefile-sensitive system with enforced 32 bits:
181
182 2b. largefile-sensitive, mpg123 with enforced _FILE_OFFSET_BITS=32
183 libmpg123: mpg123_open_32
184 lfs_alias: mpg123_open -> mpg123_open_32
185 lfs_wrap: <none>
186
187 All cases are supported with this significant change from 1.15.4:
188 Make the aliases use a defined lfs_alias_t, which can be long or off_t,
189 depending on what is the default type for offsets on the platform.
190 Folks who try _FILE_OFFSET_BITS=32 on a system that only supports
191 64 bit get a linking error during mpg123 build (from the _64 aliases),
192 which I consider to be a feature.
193
194 I salute anyone who is not confused after reading this.