Notes on post-processing and merging data files In main g0n directory (~/g0n): 1. Batches of 1000: when each batch of 1000 levels is running, run ./postproc nnn (e.g. ./postproc 399 for the range 399000-399999); this will keep looping until all are done. 2. Batches of 100 & 1000: when postproc has finished a batch of 1000, run nnn=492; for i in `seq 0 9`; do ./merge00 ${nnn}${i}; done and then (assuming no problems arise) ./merge000 nnn to create 12 files of the form g0n/data/*.nnn000-nnn999 . 3. Batches of 10000: when 10 batches of 1000 (nn0, ..., nn9) have thus been merged, run ./merge0000 nn to create 12 files of the form g0n/data/*.nn0000-nn9999 . Copy these into ~/ecdata, move them into the appropriate subdirectories, and use "git add" to add them to the git repository: nn=49 NN=${nn}0000-${nn}9999 cd ~/g0n/data for f in curves allcurves paricurves allbigsha allbsd alldegphi allgens allisog aplist count degphi intpts; do cp ${f}.${NN} ~/ecdata/${f}; done cd ~/ecdata for f in curves allcurves paricurves allbigsha allbsd alldegphi allgens allisog aplist count degphi intpts; do git add ${f}; done 4. Make alllabels file: sage: %runfile "scripts/labels.py" sage: make_alllabels("curves/curves.400000-409999") (outputs a line to screen every 1000 input lines, takes about 40s per 1000, so about 30 minutes) [quit sage] Check that outputfile (e.g. talllabels.400000-409999) & rename without the prefix "t", moving to the alllabels/ subdirectory. Add to git tracking: mv talllabels.400000-409999 alllabels/alllabels.400000-409999 git add alllabels/ From now on work in data directory (~/ecdata) which is a git repository linked to https://github.com/JohnCremona/ecdata. All files are in the master branch. GitHub automatically cretes the website at http://johncremona.github.io/ecdata/ from the docs directory in this branch. 5. Email Sutherland and ask him to run his script on any new allcurves file (e.g. allcurves.400000-409999) and rename his output galrep.* (e.g. galrep.400000-409999), move to into galrep/ and git add galrep/ 5a. Create 2adic images file and move to correct place: magma -b filename:=allcurves/allcurves.400000-409999 scripts/2adic.m mv 2adic.400000-409999 2adic/ git add 2adic/ 6. Files to be edited are: docs/table.html, docs/shas.html, Makefile, docs/index.html, docs/manin.txt, docs/release_notes.md. The first two are now created automatically by Python scripts (scripts/sharanktable.py and scripts/summarytable.py), but in each case there is a preliminary manual check to see if new Sha values have occurred or new ranks (!) in which case the scripts will need to be edited. (6a): docs/shas.html (using scripts/sharanktable.py) Check up on new Sha records using sort allbigsha/*9 -n -k 7 | tail to see if the script needs to provide extra columns for the table. Current range is s^2 for s in range(34)+[41,43,47,50,75]. Use sharanktable.py to create a new version newshas.html: sage: %runfile scripts/sharanktable.py sage: make_rankshatable(50) # to go up to 499999 If happy: mv newshas.html docs/shas.html (6b): docs/table.html (using scripts/summarytable.py) Check up on new rank records (!) using sort curves/curves.*9 -n -k 5 | tail -1 to see if the script needs adjusting (record is now 4). Use scripts/summarytable.py to create a new version newtable.html: sage: %runfile scripts/summarytable.py sage: make_table(50) # to go up to 409999 If happy: mv newtable.html docs/table.html (6c): docs/release_notes.md: Add suitable section at the top. (6d): Makefile: nothing should need doing. (6e): docs/index.html: 2 changes in lines with "pdate"; change "up to ..."; change record Sha if necessary; add an extra file in each section. Edit the paragraph "...curves with nontrivial Sha..." using cd allbigsha cat allbigsha.*9 | wc -l cat allbigsha.*9 | awk '$5==0' | wc -l cat allbigsha.*9 | awk '$5==1' | wc -l cat allbigsha.*9 | awk '$5==2' | wc -l cat allbigsha.*9 | awk '$5==3' | wc -l cat allbigsha.*9 | awk '$5==4' | wc -l cd .. (6f) docs/manin.txt needs some real work: [In g0n working dir] N=49 let 'Nminus1 = N-1' NCL=`awk '$3==1' data/allcurves.${N}0000-${N}9999 | wc -l` echo "$NCL isogeny classes in range ${N}0000-${N}9999" NCL2=`awk '$3==2' data/allcurves.${N}0000-${N}9999 | wc -l` let 'NCL1 = NCL-NCL2' echo "with ${NCL1} classes of size 1 and ${NCL2} of size at least 2" ./h1pperiods < data/allcurves.${N}0000-${N}9999 > h1pp/h1pp.long.${N} grep optimal h1pp/h1pp.long.${N} > h1pp/h1pp.out.${N} wc -l h1pp/h1pp.out.${N} # should = $NCL2 grep -c "c=1" h1pp/h1pp.out.${N} # $NCL2-this counts #classes with c=1 not known # create optimality record file: grep "opt" h1pp/h1pp.out.${N} > data/optimality.${N} cp data/optimality.${N} $HOME/ecdata/optimality/ (cd $HOME/ecdata/; git add optimality/optimality.${N}) # Display classes where c=1 not proved: grep -v "c=1" data/optimality.${N} cat h1pp/h1pp.conc.20-${Nminus1} data/optimality.${N} > h1pp/h1pp.conc.20-${N} cat h1pp/h1pp.conc.13-${Nminus1} data/optimality.${N} > h1pp/h1pp.conc.13-${N} cat h1pp/h1pp.conc.6-${Nminus1} data/optimality.${N} > h1pp/h1pp.conc.6-${N} # Numbers for lines 14-16 of manin.txt (edit the lines below): Nall=`cat ~/ecdata/curves/curves.*9 | awk '($1>200000)&&($1<500000)' | wc -l` N2=`cat ~/ecdata/allcurves/allcurves.*9 | awk '($3==2)&&($1>200000)&&($1<500000)' | wc -l` let 'N1=Nall-N2' echo $Nall $N1 $N2 # Optimality counts # classes of size > 1: wc -l h1pp/h1pp.conc.20-${N} # classes where c=1 & optimal curve known: grep -c "optimal curve is " h1pp/h1pp.conc.20-${N} # classes where c=1 known but >1 possible optimal curve: grep -c " possible " h1pp/h1pp.conc.20-${N} # classes where c=1 not known without more work: grep -v "curve is" h1pp/h1pp.conc.20-${N} | grep -v "possible" | wc -l # counts for numbers of possible optimal curves (2-6) for n in `seq 2 6`; do echo $n; grep -c "$n possible " h1pp/h1pp.conc.20-${N}; done --Now use the above numbers to manually edit docs/manin.txt. =============== 1. make commit This commits all the changes and pushes to github, so the website is updated too. 2. (a) email Bill Allombert to he can update pari's elldata [done to 500000] (b) email John Cannon so he can update Magma's database [done to 500000] (c) Update LMFDB. On atkin, make sure that lmfdb can connect to the database, then %sage sage: %runfile scripts/elliptic_curves/import_ec_data.py sage: upload_to_db("/home/jec/ecdata",380000,389999) [done to 500000] (d) Update Sage's optional spkg (this assumes that the old version of the optional spkg is already installed and that the updated ecdata repository exists): (i) Move aside the old db: cd SAGE_ROOT/local/share/cremona mv cremona.db cremona.db.bak (ii) Create the new db: sage sage: C = sage.databases.cremona.LargeCremonaDatabase('cremona',False, True) sage: C._init_from_ftpdata('ecdata/',0) sage: CDB=CremonaDatabase() sage: CDB.number_of_curves() 3064705 sage: CDB.number_of_isogeny_classes() 2164260 sage: CDB.largest_conductor() 499998 cp SAGE_ROOT/local/share/cremona/cremona.db ~/ecdata/src cd ~/ecdata chmod -R 744 src tar cvf database_cremona_ellcurve-20190911.tar src bzip2 database_cremona_ellcurve-20190911.tar chmod a+r database_cremona_ellcurve-20190911.tar.bz2 cp database_cremona_ellcurve-20190911.tar.bz2 SAGE_ROOT/upstream/ scp database_cremona_ellcurve-20190911.tar.bz2 warwick:public_html/ftp/ and now make a trac ticket for the upgrade. The only changes to be made (make a new git branch off develop first!) are $ ./sage --package create database_cremona_ellcurve --version 20190911 --tarball database_cremona_ellcurve-20190911.tar.bz2 --type optional [done to 500000]