Debian Codegraph @

Hledger/Cli/Commands/Import.txt

raw
1 import
2
3 Import new transactions from one or more data files to the main journal.
4
5 Flags:
6 --catchup just mark all transactions as already imported
7 --dry-run just show the transactions to be imported
8
9 This command detects new transactions in one or more data files
10 specified as arguments, and appends them to the main journal.
11
12 You can import from any input file format hledger supports, but
13 CSV/SSV/TSV files, downloaded from financial institutions, are the most
14 common import source.
15
16 The import destination is the default journal file, or another specified
17 in the usual way with $LEDGER_FILE or -f/--file. It should be in journal
18 format.
19
20 Examples:
21
22 $ hledger import bank1-checking.csv bank1-savings.csv
23
24 $ hledger import *.csv
25
26 Import dry run
27
28 It's useful to preview the import by running first with --dry-run, to
29 sanity check the range of dates being imported, and to check the effect
30 of your conversion rules if converting from CSV. Eg:
31
32 $ hledger import bank.csv --dry-run
33
34 The dry run output is valid journal format, so hledger can re-parse it.
35 If the output is large, you could show just the uncategorised
36 transactions like so:
37
38 $ hledger import --dry-run bank.csv | hledger -f- -I print unknown
39
40 You could also run this repeatedly to see the effect of edits to your
41 conversion rules:
42
43 $ watchexec -- "hledger import --dry-run bank.csv | hledger -f- -I print unknown"
44
45 Once the conversion and dates look good enough to import to your
46 journal, perhaps with some manual fixups to follow, you would do the
47 actual import:
48
49 $ hledger import bank.csv
50
51 Overlap detection
52
53 Reading CSV files is built in to hledger, and not specific to import; so
54 you could also import by doing hledger -f bank.csv print >>$LEDGER_FILE.
55
56 But import is easier and provides some advantages. The main one is that
57 it avoids re-importing transactions it has seen on previous runs. This
58 means you don't have to worry about overlapping data in successive
59 downloads of your bank CSV; just download and import as often as you
60 like, and only the new transactions will be imported each time.
61
62 We don't call this "deduplication", as it's generally not possible to
63 reliably detect duplicates in bank CSV. Instead, import remembers the
64 latest date processed previously in each CSV file (saving it in a hidden
65 file), and skips any records prior to that date. This works well for
66 most real-world CSV, where:
67
68 1. the data file name is stable (does not change) across imports
69 2. the item dates are stable across imports
70 3. the order of same-date items is stable across imports
71 4. the newest items have the newest dates
72
73 (Occasional violations of 2-4 are often harmless; you can reduce the
74 chance of disruption by downloading and importing more often.)
75
76 Overlap detection is automatic, and shouldn't require much attention
77 from you, except perhaps at first import (see below). But here's how it
78 works:
79
80 - For each FILE being imported from:
81
82 1. hledger reads a file named .latest.FILE file in the same
83 directory, if any. This file contains the latest record date
84 previously imported from FILE, in YYYY-MM-DD format. If multiple
85 records with that date were imported, the date is repeated on N
86 lines.
87
88 2. hledger reads records from FILE. If a latest date was found in
89 step 1, any records before that date, and the first N records on
90 that date, are skipped.
91
92 - After a successful import from all FILEs, without error and without
93 --dry-run, hledger updates each FILE's .latest.FILE for next time.
94
95 If this goes wrong, it's relatively easy to repair:
96
97 - You'll notice it before import when you preview with import --dry-run.
98 - Or after import when you try to reconcile your hledger account
99 balances with your bank.
100 - hledger print -f FILE.csv will show all recently downloaded
101 transactions. Compare these with your journal. Copy/paste if needed.
102 - Update your conversion rules and print again, if needed.
103 - You can manually update or remove the .latest file, or use
104 import --catchup FILE.
105 - Download and import more often, eg twice a week, at least while you
106 are learning. It's easier to review and troubleshoot when there are
107 fewer transactions.
108
109 First import
110
111 The first time you import from a file, when no corresponding .latest
112 file has been created yet, all of the records will be imported.
113
114 But perhaps you have been entering the data manually, so you know that
115 all of these transactions are already recorded in the journal. In this
116 case you can run hledger import --catchup once. This will create a
117 .latest file containing the latest CSV record date, so that none of
118 those records will be re-imported.
119
120 Or, if you know that some but not all of the transactions are in the
121 journal, you can create the .latest file yourself. Eg, let's say you
122 previously recorded foobank transactions up to 2024-10-31 in the
123 journal. Then in the directory where you'll be saving foobank.csv, you
124 would create a .latest.foobank.csv file containing
125
126 2024-10-31
127
128 Or if you had three foobank transactions recorded with that date, you
129 would repeat the date that many times:
130
131 2024-10-31
132 2024-10-31
133 2024-10-31
134
135 Then hledger import foobank.csv [--dry-run] will import only the newer
136 records.
137
138 Importing balance assignments
139
140 Journal entries added by import will have all posting amounts made
141 explicit (like print -x).
142
143 This means that any balance assignments in the imported entries would
144 need to be evaluated. But this generally isn't possible, as the main
145 file's account balances are not visible during import. So try to avoid
146 generating balance assignments with your CSV rules, or importing from a
147 journal that contains balance assignments. (Balance assignments are best
148 avoided anyway.)
149
150 But if you must use them, eg because your CSV includes only balances:
151 you can import with print, which leaves implicit amounts implicit.
152 (print can also do overlap detection like import, with the --new flag):
153
154 $ hledger print --new -f bank.csv >> $LEDGER_FILE
155
156 (If you think import should preserve implicit balances, please test that
157 and send a pull request.)
158
159 Import and commodity styles
160
161 Amounts in entries added by import will be formatted according to the
162 journal's canonical commodity styles, as declared by commodity
163 directives or inferred from the journal's amounts.
164
165 Related: CSV > Amount decimal places.
166
167 Import archiving
168
169 When importing from a CSV rules file (hledger import bank.rules), you
170 can use the archive rule to enable automatic archiving of the data file.
171 After a successful import, the data file (specified by source) will be
172 moved to an archive folder (data/, next to the rules file,
173 auto-created), and renamed similar to the rules file, with a date. This
174 can be useful for troubleshooting, detecting variations in your banks'
175 CSV data, regenerating entries with improved rules, etc.
176
177 The archive rule also causes import to handle source glob patterns
178 differently: when there are multiple matched files, it will pick the
179 oldest, not the newest.
180
181 Import special cases
182
183 Deduplication
184
185 Here are two kinds of "deduplication" which import does not handle (and
186 should not, because these can happen legitimately in financial data):
187
188 - Two or more of the new CSV records are identical, and generate
189 identical new journal entries.
190 - A new CSV record generates a journal entry identical to one(s) already
191 in the journal.
192
193 Varying file name
194
195 If you have a download whose file name varies, you could rename it to a
196 fixed name after each download. Or you could use a CSV source rule with
197 a suitable glob pattern, and import from the .rules file.
198
199 Multiple versions
200
201 Say you download bank.csv, import it, but forget to delete it from your
202 downloads folder. The next time you download it, your web browser will
203 save it as (eg) bank (2).csv. The source rule's glob patterns are for
204 just this situation: instead of specifying source bank.csv, specify
205 source bank*.csv. Then hledger -f bank.rules CMD or
206 hledger import bank.rules will automatically pick the newest matched
207 file (bank (2).csv).
208
209 Alternately, what if you download, but forget to import or delete, then
210 download again ? Now each of bank.csv and bank (2).csv might contain
211 data that's not in the other, and not in your journal. In this case,
212 it's best to import each of them in turn, oldest first (otherwise,
213 overlap detection could cause new records to be skipped). Enabling
214 import archiving ensures this. Then
215 hledger import bank.rules; hledger import bank.rules will import and
216 archive first bank.csv, then bank (2).csv.