| 1 |
import |
| 2 |
|
| 3 |
Import new transactions from one or more data files to the main journal. |
| 4 |
|
| 5 |
Flags: |
| 6 |
--catchup just mark all transactions as already imported |
| 7 |
--dry-run just show the transactions to be imported |
| 8 |
|
| 9 |
This command detects new transactions in one or more data files |
| 10 |
specified as arguments, and appends them to the main journal. |
| 11 |
|
| 12 |
You can import from any input file format hledger supports, but |
| 13 |
CSV/SSV/TSV files, downloaded from financial institutions, are the most |
| 14 |
common import source. |
| 15 |
|
| 16 |
The import destination is the default journal file, or another specified |
| 17 |
in the usual way with $LEDGER_FILE or -f/--file. It should be in journal |
| 18 |
format. |
| 19 |
|
| 20 |
Examples: |
| 21 |
|
| 22 |
$ hledger import bank1-checking.csv bank1-savings.csv |
| 23 |
|
| 24 |
$ hledger import *.csv |
| 25 |
|
| 26 |
Import dry run |
| 27 |
|
| 28 |
It's useful to preview the import by running first with --dry-run, to |
| 29 |
sanity check the range of dates being imported, and to check the effect |
| 30 |
of your conversion rules if converting from CSV. Eg: |
| 31 |
|
| 32 |
$ hledger import bank.csv --dry-run |
| 33 |
|
| 34 |
The dry run output is valid journal format, so hledger can re-parse it. |
| 35 |
If the output is large, you could show just the uncategorised |
| 36 |
transactions like so: |
| 37 |
|
| 38 |
$ hledger import --dry-run bank.csv | hledger -f- -I print unknown |
| 39 |
|
| 40 |
You could also run this repeatedly to see the effect of edits to your |
| 41 |
conversion rules: |
| 42 |
|
| 43 |
$ watchexec -- "hledger import --dry-run bank.csv | hledger -f- -I print unknown" |
| 44 |
|
| 45 |
Once the conversion and dates look good enough to import to your |
| 46 |
journal, perhaps with some manual fixups to follow, you would do the |
| 47 |
actual import: |
| 48 |
|
| 49 |
$ hledger import bank.csv |
| 50 |
|
| 51 |
Overlap detection |
| 52 |
|
| 53 |
Reading CSV files is built in to hledger, and not specific to import; so |
| 54 |
you could also import by doing hledger -f bank.csv print >>$LEDGER_FILE. |
| 55 |
|
| 56 |
But import is easier and provides some advantages. The main one is that |
| 57 |
it avoids re-importing transactions it has seen on previous runs. This |
| 58 |
means you don't have to worry about overlapping data in successive |
| 59 |
downloads of your bank CSV; just download and import as often as you |
| 60 |
like, and only the new transactions will be imported each time. |
| 61 |
|
| 62 |
We don't call this "deduplication", as it's generally not possible to |
| 63 |
reliably detect duplicates in bank CSV. Instead, import remembers the |
| 64 |
latest date processed previously in each CSV file (saving it in a hidden |
| 65 |
file), and skips any records prior to that date. This works well for |
| 66 |
most real-world CSV, where: |
| 67 |
|
| 68 |
1. the data file name is stable (does not change) across imports |
| 69 |
2. the item dates are stable across imports |
| 70 |
3. the order of same-date items is stable across imports |
| 71 |
4. the newest items have the newest dates |
| 72 |
|
| 73 |
(Occasional violations of 2-4 are often harmless; you can reduce the |
| 74 |
chance of disruption by downloading and importing more often.) |
| 75 |
|
| 76 |
Overlap detection is automatic, and shouldn't require much attention |
| 77 |
from you, except perhaps at first import (see below). But here's how it |
| 78 |
works: |
| 79 |
|
| 80 |
- For each FILE being imported from: |
| 81 |
|
| 82 |
1. hledger reads a file named .latest.FILE file in the same |
| 83 |
directory, if any. This file contains the latest record date |
| 84 |
previously imported from FILE, in YYYY-MM-DD format. If multiple |
| 85 |
records with that date were imported, the date is repeated on N |
| 86 |
lines. |
| 87 |
|
| 88 |
2. hledger reads records from FILE. If a latest date was found in |
| 89 |
step 1, any records before that date, and the first N records on |
| 90 |
that date, are skipped. |
| 91 |
|
| 92 |
- After a successful import from all FILEs, without error and without |
| 93 |
--dry-run, hledger updates each FILE's .latest.FILE for next time. |
| 94 |
|
| 95 |
If this goes wrong, it's relatively easy to repair: |
| 96 |
|
| 97 |
- You'll notice it before import when you preview with import --dry-run. |
| 98 |
- Or after import when you try to reconcile your hledger account |
| 99 |
balances with your bank. |
| 100 |
- hledger print -f FILE.csv will show all recently downloaded |
| 101 |
transactions. Compare these with your journal. Copy/paste if needed. |
| 102 |
- Update your conversion rules and print again, if needed. |
| 103 |
- You can manually update or remove the .latest file, or use |
| 104 |
import --catchup FILE. |
| 105 |
- Download and import more often, eg twice a week, at least while you |
| 106 |
are learning. It's easier to review and troubleshoot when there are |
| 107 |
fewer transactions. |
| 108 |
|
| 109 |
First import |
| 110 |
|
| 111 |
The first time you import from a file, when no corresponding .latest |
| 112 |
file has been created yet, all of the records will be imported. |
| 113 |
|
| 114 |
But perhaps you have been entering the data manually, so you know that |
| 115 |
all of these transactions are already recorded in the journal. In this |
| 116 |
case you can run hledger import --catchup once. This will create a |
| 117 |
.latest file containing the latest CSV record date, so that none of |
| 118 |
those records will be re-imported. |
| 119 |
|
| 120 |
Or, if you know that some but not all of the transactions are in the |
| 121 |
journal, you can create the .latest file yourself. Eg, let's say you |
| 122 |
previously recorded foobank transactions up to 2024-10-31 in the |
| 123 |
journal. Then in the directory where you'll be saving foobank.csv, you |
| 124 |
would create a .latest.foobank.csv file containing |
| 125 |
|
| 126 |
2024-10-31 |
| 127 |
|
| 128 |
Or if you had three foobank transactions recorded with that date, you |
| 129 |
would repeat the date that many times: |
| 130 |
|
| 131 |
2024-10-31 |
| 132 |
2024-10-31 |
| 133 |
2024-10-31 |
| 134 |
|
| 135 |
Then hledger import foobank.csv [--dry-run] will import only the newer |
| 136 |
records. |
| 137 |
|
| 138 |
Importing balance assignments |
| 139 |
|
| 140 |
Journal entries added by import will have all posting amounts made |
| 141 |
explicit (like print -x). |
| 142 |
|
| 143 |
This means that any balance assignments in the imported entries would |
| 144 |
need to be evaluated. But this generally isn't possible, as the main |
| 145 |
file's account balances are not visible during import. So try to avoid |
| 146 |
generating balance assignments with your CSV rules, or importing from a |
| 147 |
journal that contains balance assignments. (Balance assignments are best |
| 148 |
avoided anyway.) |
| 149 |
|
| 150 |
But if you must use them, eg because your CSV includes only balances: |
| 151 |
you can import with print, which leaves implicit amounts implicit. |
| 152 |
(print can also do overlap detection like import, with the --new flag): |
| 153 |
|
| 154 |
$ hledger print --new -f bank.csv >> $LEDGER_FILE |
| 155 |
|
| 156 |
(If you think import should preserve implicit balances, please test that |
| 157 |
and send a pull request.) |
| 158 |
|
| 159 |
Import and commodity styles |
| 160 |
|
| 161 |
Amounts in entries added by import will be formatted according to the |
| 162 |
journal's canonical commodity styles, as declared by commodity |
| 163 |
directives or inferred from the journal's amounts. |
| 164 |
|
| 165 |
Related: CSV > Amount decimal places. |
| 166 |
|
| 167 |
Import archiving |
| 168 |
|
| 169 |
When importing from a CSV rules file (hledger import bank.rules), you |
| 170 |
can use the archive rule to enable automatic archiving of the data file. |
| 171 |
After a successful import, the data file (specified by source) will be |
| 172 |
moved to an archive folder (data/, next to the rules file, |
| 173 |
auto-created), and renamed similar to the rules file, with a date. This |
| 174 |
can be useful for troubleshooting, detecting variations in your banks' |
| 175 |
CSV data, regenerating entries with improved rules, etc. |
| 176 |
|
| 177 |
The archive rule also causes import to handle source glob patterns |
| 178 |
differently: when there are multiple matched files, it will pick the |
| 179 |
oldest, not the newest. |
| 180 |
|
| 181 |
Import special cases |
| 182 |
|
| 183 |
Deduplication |
| 184 |
|
| 185 |
Here are two kinds of "deduplication" which import does not handle (and |
| 186 |
should not, because these can happen legitimately in financial data): |
| 187 |
|
| 188 |
- Two or more of the new CSV records are identical, and generate |
| 189 |
identical new journal entries. |
| 190 |
- A new CSV record generates a journal entry identical to one(s) already |
| 191 |
in the journal. |
| 192 |
|
| 193 |
Varying file name |
| 194 |
|
| 195 |
If you have a download whose file name varies, you could rename it to a |
| 196 |
fixed name after each download. Or you could use a CSV source rule with |
| 197 |
a suitable glob pattern, and import from the .rules file. |
| 198 |
|
| 199 |
Multiple versions |
| 200 |
|
| 201 |
Say you download bank.csv, import it, but forget to delete it from your |
| 202 |
downloads folder. The next time you download it, your web browser will |
| 203 |
save it as (eg) bank (2).csv. The source rule's glob patterns are for |
| 204 |
just this situation: instead of specifying source bank.csv, specify |
| 205 |
source bank*.csv. Then hledger -f bank.rules CMD or |
| 206 |
hledger import bank.rules will automatically pick the newest matched |
| 207 |
file (bank (2).csv). |
| 208 |
|
| 209 |
Alternately, what if you download, but forget to import or delete, then |
| 210 |
download again ? Now each of bank.csv and bank (2).csv might contain |
| 211 |
data that's not in the other, and not in your journal. In this case, |
| 212 |
it's best to import each of them in turn, oldest first (otherwise, |
| 213 |
overlap detection could cause new records to be skipped). Enabling |
| 214 |
import archiving ensures this. Then |
| 215 |
hledger import bank.rules; hledger import bank.rules will import and |
| 216 |
archive first bank.csv, then bank (2).csv. |