| 1 |
This documents the effort to make dmapd and libdmapsharing more memory |
| 2 |
efficient. This document describes work done in both libdmapsharing and |
| 3 |
dmapd, because dmapd is the primary application of libdmapsharing's |
| 4 |
server-side functionality. |
| 5 |
|
| 6 |
I found that dmapd used a significant amount of heap space while trying to |
| 7 |
port dmapd to OpenWRT on a WRT160NL router with 32 Megabytes of RAM. A |
| 8 |
2,419-song library caused dmapd on x86_64 to use 7,413,760 bytes of heap |
| 9 |
space after reading the library into memory. Furthermore, connecting to |
| 10 |
dmapd using a DAAP client and receiving a list of songs caused heap usage |
| 11 |
to increase to over 10 Megabytes. Note that this memory usage might be |
| 12 |
smaller on the WRT160NL due to its 32-bit architecture, but it is still |
| 13 |
significant in size. |
| 14 |
|
| 15 |
After attempting to optimize memory use, I measured the following on |
| 16 |
OpenWrt / WRT160NL (*** BUT WITH NO METADATA READ! ***): |
| 17 |
|
| 18 |
Heap, after music is loaded: 0xf5000 (1,003,520) |
| 19 |
Heap, after client received song list: 0x926000 (9,592,832) |
| 20 |
|
| 21 |
SOME METADATA (MP3 ONLY): |
| 22 |
|
| 23 |
Heap, after music is loaded: 0x2d3000 (2,961,408) |
| 24 |
Heap, after client received song list: 0xafa000 (11,509,760) |
| 25 |
|
| 26 |
I use the following tools to troubleshoot memory issues: |
| 27 |
|
| 28 |
grep heap /proc/PID/maps (current heap size) |
| 29 |
valgrind / memcheck (memory leaks) |
| 30 |
valgrind / massif (max heap size and memory over time) |
| 31 |
gdb |
| 32 |
objdump -t OBJECT-FILE | grep -e "\.bss" -e "\.data" | sort -k5 |
| 33 |
| c++filt | tail |
| 34 |
|
| 35 |
I have made the following changes in order to make dmapd more memory |
| 36 |
efficient: |
| 37 |
|
| 38 |
1. Replace the hash table-based DMAPDb implementation with a Berkeley |
| 39 |
Database implementation. Currently measuring the memory use compared to |
| 40 |
the in-memory database. |
| 41 |
|
| 42 |
2. Fixed several memory leaks after analyzing with valgrind. |
| 43 |
|
| 44 |
Heap without client connection: 3,940,352 bytes |
| 45 |
|
| 46 |
3. Implement database cache so that dmapd does not need to use GStreamer |
| 47 |
to read media metadata each time it is run. |
| 48 |
|
| 49 |
Heap without client connection using DB cache: 3,346,432 bytes |
| 50 |
|
| 51 |
4. Replace the DMAPDb in the DMAPContainerRecord implementation with a |
| 52 |
GSList of ID's. |
| 53 |
|
| 54 |
Heap without client connection using DB cache: 4,759,552 bytes |
| 55 |
|
| 56 |
5. Implement "stringletons" and fix errors in 4) above. |
| 57 |
|
| 58 |
Heap without client connection using DB cache: 2,854,912 bytes |
| 59 |
Heap without client connection: 3,518,464 bytes |
| 60 |
Heap after client connection and song list: 6,864,896 bytes |
| 61 |
Max heap usages (massif): 8,945,760 bytes |
| 62 |
|
| 63 |
6. Modify libdmapsharing so that it responds to "/1/items" requests in |
| 64 |
chunks, avoiding the need to build the entire response in memory. |
| 65 |
|
| 66 |
The effectiveness of this can be measured by observing the |
| 67 |
maximum heap usage after a client has asked for a full media |
| 68 |
list. Previous to this change, dmapd would use 7,811,072 bytes |
| 69 |
of heap to describe a 2,232-song library. After this change, |
| 70 |
dmapd's usage was 3,665,920 bytes, a very small increase over |
| 71 |
the memory used by dmapd immediately after starting. |
| 72 |
|
| 73 |
7. Write a disk-based database for dmapd. |
| 74 |
|
| 75 |
In order to minimize the memory used by a newly started dmapd, |
| 76 |
I wrote a disk-based DMAPDb backend. This backend stores all |
| 77 |
database data in files and avoids storing the entire database |
| 78 |
in memory. To measure the effectiveness, I measured the memory |
| 79 |
usage of dmapd after loading a 3,276-photograph library (note |
| 80 |
that thumbnails are especially problematic). The GHashTable-based |
| 81 |
database used 35,885,056 bytes of heap space. The disk-based |
| 82 |
database used 4,763,648 bytes. |
| 83 |
|
| 84 |
8. Decompressing multiple scan JPEG's. |
| 85 |
|
| 86 |
VIPS must decompress multiple scan JPEG's fully in memory due |
| 87 |
to the way libjpeg works. Either 1) use embedded EXIF thumbnail |
| 88 |
or 2) skip. |