Debian Codegraph @

README-Memory

raw
1 This documents the effort to make dmapd and libdmapsharing more memory
2 efficient. This document describes work done in both libdmapsharing and
3 dmapd, because dmapd is the primary application of libdmapsharing's
4 server-side functionality.
5
6 I found that dmapd used a significant amount of heap space while trying to
7 port dmapd to OpenWRT on a WRT160NL router with 32 Megabytes of RAM. A
8 2,419-song library caused dmapd on x86_64 to use 7,413,760 bytes of heap
9 space after reading the library into memory. Furthermore, connecting to
10 dmapd using a DAAP client and receiving a list of songs caused heap usage
11 to increase to over 10 Megabytes. Note that this memory usage might be
12 smaller on the WRT160NL due to its 32-bit architecture, but it is still
13 significant in size.
14
15 After attempting to optimize memory use, I measured the following on
16 OpenWrt / WRT160NL (*** BUT WITH NO METADATA READ! ***):
17
18 Heap, after music is loaded: 0xf5000 (1,003,520)
19 Heap, after client received song list: 0x926000 (9,592,832)
20
21 SOME METADATA (MP3 ONLY):
22
23 Heap, after music is loaded: 0x2d3000 (2,961,408)
24 Heap, after client received song list: 0xafa000 (11,509,760)
25
26 I use the following tools to troubleshoot memory issues:
27
28 grep heap /proc/PID/maps (current heap size)
29 valgrind / memcheck (memory leaks)
30 valgrind / massif (max heap size and memory over time)
31 gdb
32 objdump -t OBJECT-FILE | grep -e "\.bss" -e "\.data" | sort -k5
33 | c++filt | tail
34
35 I have made the following changes in order to make dmapd more memory
36 efficient:
37
38 1. Replace the hash table-based DMAPDb implementation with a Berkeley
39 Database implementation. Currently measuring the memory use compared to
40 the in-memory database.
41
42 2. Fixed several memory leaks after analyzing with valgrind.
43
44 Heap without client connection: 3,940,352 bytes
45
46 3. Implement database cache so that dmapd does not need to use GStreamer
47 to read media metadata each time it is run.
48
49 Heap without client connection using DB cache: 3,346,432 bytes
50
51 4. Replace the DMAPDb in the DMAPContainerRecord implementation with a
52 GSList of ID's.
53
54 Heap without client connection using DB cache: 4,759,552 bytes
55
56 5. Implement "stringletons" and fix errors in 4) above.
57
58 Heap without client connection using DB cache: 2,854,912 bytes
59 Heap without client connection: 3,518,464 bytes
60 Heap after client connection and song list: 6,864,896 bytes
61 Max heap usages (massif): 8,945,760 bytes
62
63 6. Modify libdmapsharing so that it responds to "/1/items" requests in
64 chunks, avoiding the need to build the entire response in memory.
65
66 The effectiveness of this can be measured by observing the
67 maximum heap usage after a client has asked for a full media
68 list. Previous to this change, dmapd would use 7,811,072 bytes
69 of heap to describe a 2,232-song library. After this change,
70 dmapd's usage was 3,665,920 bytes, a very small increase over
71 the memory used by dmapd immediately after starting.
72
73 7. Write a disk-based database for dmapd.
74
75 In order to minimize the memory used by a newly started dmapd,
76 I wrote a disk-based DMAPDb backend. This backend stores all
77 database data in files and avoids storing the entire database
78 in memory. To measure the effectiveness, I measured the memory
79 usage of dmapd after loading a 3,276-photograph library (note
80 that thumbnails are especially problematic). The GHashTable-based
81 database used 35,885,056 bytes of heap space. The disk-based
82 database used 4,763,648 bytes.
83
84 8. Decompressing multiple scan JPEG's.
85
86 VIPS must decompress multiple scan JPEG's fully in memory due
87 to the way libjpeg works. Either 1) use embedded EXIF thumbnail
88 or 2) skip.