Debian Codegraph @

homepage/script-docs/stag-storenode.html

raw
1 <HTML>
2 <HEAD>
3 <TITLE>stag-storenode</TITLE>
4 <LINK REV="made" HREF="mailto:[email protected]">
5 </HEAD>
6
7 <BODY>
8
9 <A NAME="__index__"></A>
10 <!-- INDEX BEGIN -->
11
12 <UL>
13
14 <LI><A HREF="#name">NAME</A></LI>
15 <LI><A HREF="#synopsis">SYNOPSIS</A></LI>
16 <LI><A HREF="#description">DESCRIPTION</A></LI>
17 <UL>
18
19 <LI><A HREF="#arguments">ARGUMENTS</A></LI>
20 <UL>
21
22 <LI><A HREF="#d dbname">-d <STRONG>DBNAME</STRONG></A></LI>
23 <LI><A HREF="#u unit">-u <STRONG>UNIT</STRONG></A></LI>
24 <LI><A HREF="#c stagmapfile">-c <STRONG>STAGMAPFILE</STRONG></A></LI>
25 <LI><A HREF="#p parser">-p <STRONG>PARSER</STRONG></A></LI>
26 <LI><A HREF="#t transformer">-t <STRONG>TRANSFORMER</STRONG></A></LI>
27 <LI><A HREF="#noupdate nodelist">-noupdate <STRONG>NODELIST</STRONG></A></LI>
28 <LI><A HREF="#trust_ids">-trust_ids</A></LI>
29 </UL>
30
31 </UL>
32
33 <LI><A HREF="#making database from xml files">MAKING DATABASE FROM XML FILES</A></LI>
34 <UL>
35
36 <LI><A HREF="#step 1: prepare input file">Step 1: Prepare input file</A></LI>
37 <LI><A HREF="#step 2: generating create table statements">Step 2: Generating CREATE TABLE statements</A></LI>
38 <LI><A HREF="#step 3: store the data in the db">Step 3: Store the data in the db</A></LI>
39 </UL>
40
41 </UL>
42 <!-- INDEX END -->
43
44 <HR>
45 <P>
46 <H1><A NAME="name">NAME</A></H1>
47 <P>stag-storenode.pl</P>
48 <P>
49 <HR>
50 <H1><A NAME="synopsis">SYNOPSIS</A></H1>
51 <PRE>
52 stag-storenode.pl -d &quot;dbi:Pg:dbname=mydb;host=localhost&quot; myfile.xml</PRE>
53 <P>
54 <HR>
55 <H1><A NAME="description">DESCRIPTION</A></H1>
56 <P>This script is for storing data (specified in a nested file format
57 such as XML or S-Expressions) in a database. It assumes a database
58 schema corresponding to the tags in the input data already exists.</P>
59 <P>
60 <H2><A NAME="arguments">ARGUMENTS</A></H2>
61 <P>
62 <H3><A NAME="d dbname">-d <STRONG>DBNAME</STRONG></A></H3>
63 <P>This is either a DBI locator or the logical name of a database in the
64 DBSTAG_DBIMAP_FILE config file</P>
65 <P>
66 <H3><A NAME="u unit">-u <STRONG>UNIT</STRONG></A></H3>
67 <P>This is the node/element name on which to load; a database loading
68 event will be fired every time one of these elements is parsed; this
69 also constitutes a whole transaction</P>
70 <P>
71 <H3><A NAME="c stagmapfile">-c <STRONG>STAGMAPFILE</STRONG></A></H3>
72 <P>This is a stag mapping file, indicating which elements are aliases</P>
73 <P>
74 <H3><A NAME="p parser">-p <STRONG>PARSER</STRONG></A></H3>
75 <P>Default is xml; can be any stag compatible parser, OR a perl module
76 which will parse the input file and fire stag events (see
77 <A HREF="./Data/Stag/BaseGenerator.html">the Data::Stag::BaseGenerator manpage</A>)</P>
78 <P>
79 <H3><A NAME="t transformer">-t <STRONG>TRANSFORMER</STRONG></A></H3>
80 <P>This is the name of a perl module that will perform a transformation
81 on the stag events/XML. See also <EM>stag-handle.pl</EM></P>
82 <P>
83 <H3><A NAME="noupdate nodelist">-noupdate <STRONG>NODELIST</STRONG></A></H3>
84 <P>A comma-seperated (no spaces) list of nodes/elements on which no
85 update should be performed if a unique key is found to be present in
86 the DB</P>
87 <P>
88 <H3><A NAME="trust_ids">-trust_ids</A></H3>
89 <P>If this flag is present, the values for primary key values are
90 trusted; otherwise they are assumed to be surrogate internal IDs that
91 should not be used. In this case they will be remapped.</P>
92 <P>
93 <HR>
94 <H1><A NAME="making database from xml files">MAKING DATABASE FROM XML FILES</A></H1>
95 <P>It is possible to automatically generate a database schema and
96 populate it directly from XML files (or from Stag objects or other
97 Stag compatible files). Of course, this is no substitute for proper
98 relational design, but often it can be necessary to quickly generate
99 databases from heterogeneous XML data sources, for the purposes of
100 data mining.</P>
101 <P>There are 3 steps involved:</P>
102 <P>1. Prepare the input XML (for instance, modifying db reserved words).
103 2. Autogenerate the CREATE TABLE statements, and make a db from these.
104 3. Store the XML data in the database.</P>
105 <P>
106 <H2><A NAME="step 1: prepare input file">Step 1: Prepare input file</A></H2>
107 <P>You may need to make modifications to your XML before it can be used
108 to make a schema. If your XML elements contain any words that are
109 reserved by your DB you should change these.</P>
110 <P>Any XML processing tool (eg XSLT) can be used. Alternatively you can
111 use the script 'stag-mogrify'</P>
112 <P>e.g. to get rid of '-' characters (this is how Stag treates
113 attributes) and to change the element with postgresql reserved word
114 'date', do this:</P>
115 <PRE>
116 stag-mogrify.pl -xml -r 's/^date$/moddate/' -r 's/\-//g' data.xml &gt; data.mog.xml</PRE>
117 <P>You may also need to explicitly make elements where you will need
118 linking tables. For instance, if the relationship between 'movie' and
119 'star' is many-to-many, and your input data looks like this:</P>
120 <PRE>
121 (movie
122 (name &quot;star wars&quot;)
123 (star
124 (name &quot;mark hamill&quot;)))</PRE>
125 <P>You will need to *interpose* an element between these two, like this:</P>
126 <PRE>
127 (movie
128 (name &quot;star wars&quot;)
129 (movie2star
130 (star
131 (name &quot;mark hamill&quot;))))</PRE>
132 <P>you can do this with the -i switch:</P>
133 <PRE>
134 stag-mogrify.pl -xml -i movie,star,movie2star data.xml &gt; data.mog.xml</PRE>
135 <P>or if you simply do:</P>
136 <PRE>
137 stag-mogrify.pl -xml -i star data.xml &gt; data.mog.xml</PRE>
138 <P>the mogrifier will simply interpose an element above every time it
139 sees 'star'; the naming rule is to use the two elements with an
140 underscore between (in this case, 'movie_star').</P>
141 <P>
142 <H2><A NAME="step 2: generating create table statements">Step 2: Generating CREATE TABLE statements</A></H2>
143 <P>Use the stag-autoddl.pl script;</P>
144 <PRE>
145 stag-autoddl.pl data.mog.xml &gt; table.sql</PRE>
146 <P>The default rule is to create foreign keys from the nested element to
147 the outer element; you will want linking tables tobe treated
148 differently (a linking table will point to parent and child elements).</P>
149 <PRE>
150 stag-autoddl.pl -l movie2star -l star2character data.mog.xml &gt; table.sql</PRE>
151 <P>Once you have done this, load the statements into your db; eg for postgresql
152 (for other databases, use <A HREF="./SQL/Translator.html">the SQL::Translator manpage</A>)</P>
153 <PRE>
154 psql -a mydb &lt; table.sql</PRE>
155 <P>If something goes wrong, go back to step 1 and sort it out!</P>
156 <P>Note that certain rules are followed: ever table generated gets a
157 surrogate primary key of type 'serial'; this is used to generate
158 foreign key relationships. The rule used is primary and foreign key
159 names are the name of the table with the '_id' suffix.</P>
160 <P>Feel free to modify the autogenerated schema at this stage (eg add
161 uniqueness constraints)</P>
162 <P>
163 <H2><A NAME="step 3: store the data in the db">Step 3: Store the data in the db</A></H2>
164 <PRE>
165 stag-storenode.pl -u movie -d 'dbi:Pg:mydb' data.mog.xml</PRE>
166 <P>You generally dont need extra metadata here; everything can be
167 infered by introspecting the database.</P>
168 <P>The -u|unit switch controls when transactions are committed</P>
169 <P>If this works, you should now be able to retreive XML from the database, eg</P>
170 <PRE>
171 selectall_xml.pl -d 'dbi:Pg:mydb' 'SELECT * FROM x NATURAL JOIN y'</PRE>
172
173 </BODY>
174
175 </HTML>