User Tools

  • Logged in as: anonymous (anonymous)
  • Log Out

Site Tools


mantisbt:importexport

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
mantisbt:importexport [2008/03/01 09:46] giallumantisbt:importexport [2024/03/26 05:03] (current) – [Other] formatting xmllint code block dregad
Line 2: Line 2:
  
    * **Author**: Gianluca Sforna (giallu)    * **Author**: Gianluca Sforna (giallu)
-   * **Status**: Draft | <del>Work in Progress</del> | <del>Implemented</del>+   * **Status**: <del>Draft</del> | <del>Work in Progress</del> | Implemented
    * **Associated Issue**: http://www.mantisbt.org/bugs/view.php?id=4220    * **Associated Issue**: http://www.mantisbt.org/bugs/view.php?id=4220
 +
  
 ===== Introduction ===== ===== Introduction =====
 A complete solution for copying a list of issues between two different Mantis instances is something asked from quite some time: A complete solution for copying a list of issues between two different Mantis instances is something asked from quite some time:
  
-http://www.mantisbt.org/bugs/view.php?id=4220+  * http://www.mantisbt.org/bugs/view.php?id=4220 
 +  * http://www.mantisbt.org/bugs/view.php?id=5310
  
-For a greater flexibility, the use of an XML format looks like the best option (XML export was also asked in http://www.mantisbt.org/bugs/view.php?id=4024)+For a greater flexibility, the use of an XML format looks like the best option (XML export was also asked in http://www.mantisbt.org/bugs/view.php?id=4024).
  
  
Line 20: Line 22:
   * Activate the Export command   * Activate the Export command
   * An XML file is produced and downloaded   * An XML file is produced and downloaded
-  * Log in instance B as administrator+  * Log in instance B as project manager
   * Import the XML file and watch instance B populated with new bugs   * Import the XML file and watch instance B populated with new bugs
 +
 +The first implementation will support all the issue fields (including extended info) and custom fields.
 +
 +Given the extensible nature of the XML format, this feature can be enhanced later to support notes, attachments and all other informations related to the issue.
 +
 +
  
 ===== Design choices and issues ===== ===== Design choices and issues =====
  
-The main hurdle for this feature is to define the behavior when any piece of information is internal reference to instance A.+The main hurdle for this feature is to define the behaviour when any piece of information is an internal reference to instance A.
  
 This includes: This includes:
-  Projects - the selected issues could be from more than one project in instance A and there is no guarantee the same project(s) will be in instance B +=== Project IDs=== 
-    assume the import will be in a single project, so warn the user on export + 
-  * User IDs +** Issue ** 
-  + 
 +the selected issues could be from more than one project in instance A and there is no guarantee the same project(s) will be in instance B 
 + 
 +** Proposed solution ** 
 + 
 +import will happen on a single project in instance B 
 + 
 +** Status ** 
 + 
 +DONE 
 + 
 + 
 +=== User IDs === 
 + 
 +** Issue **  
 + 
 +user ids will not match between the instances 
 + 
 +** Proposed solution ** 
 + 
 +On import pageask what to do with users. Options are: 
 +  * squash all ids to a single one in instance B 
 +  * attempt to find a matching user by username, then fall back to squashing id 
 + 
 +** Status ** 
 + 
 +DONE 
 +A function implementing the above behaviour exists, but composing a list of users is memory intensive (more than 32Mb to execute with the mantis DB) so I'm refraining from using it. 
 + 
 +The current implementation tries by default to match usernames, then squash to the userID performing the import operation 
 + 
 + 
 +=== Bug and Notes internal links === 
 +** Issue **  
 + 
 +Notes and bugs can contain references to other notes or bugs (by default they are made with the syntax #bugId and ~noteId) 
 + 
 +** Proposed solution ** 
 + 
 +The references to bugs and notes that are imported in the same transaction can be rewrote to reflect the new id assigned in instance B. 
 + 
 +On import page, allow choosing whether to try converting references and one of the following default actions: 
 +  * Replace internal links with links to original items in instance A 
 +  * Replace them with fake links, pretending the linked bug or note was deleted 
 + 
 +** Status ** 
 + 
 +DONE (notes are not in the initial implementation, but code can support those) 
 + 
 + 
 + 
 +=== Categories === 
 +** Issue **  
 + 
 +Category is a required field, but nothing guarantees the target project on instance B will use the same ones as instance A. 
 + 
 +** Proposed solution ** 
 + 
 +Add import options: 
 +  * skip import of issues with unknown categories 
 +  * create missing categories (dangerous...) 
 + 
 +** Status ** 
 + 
 +PENDING 
 + 
 + 
 +===== Metadata ==== 
 + 
 +In order to support all the options above, some additional information will need to be included in the exported file. 
 + 
 +  * Full URL ($g_path) of instance A 
 +  * The bug link trigger ( $g_bug_link_tag ) 
 +  * The note link trigger ( $g_bugnote_link_tag )
  
  
 ===== Action Items ===== ===== Action Items =====
-  
  
 +==== Phase 1 ====
 +  * Prepare a basic DTD for the XML file (just bug data)
 +  * Prepare a sample XML file
 +  * Add "XML Export" link to view_all_bug_page.php
 +  * Add a new link in the manage menu to manage_import_issues_page.php
 +  * Create the manage_import_issues_page.php page
 +
 +==== Phase 2 ====
 +  * Add bug notes to the DTD
 +
 +==== Import Issues Page details ====
 +The import strategy will be:
 +  * STEP 1: let the manager upload the input file
 +  * STEP 2: ask what to do with user IDs and bug(note) links as drafted above
 +  * STEP 3: perform the operation
 +
 +=== STEP 3 details ===
 +<code>
 +  open XML file
 +  initialize oldID => newID array for: issues, notes
 +  for each <issue> element
 +    create a new Bugdata object
 +    populate data with <issue> fields
 +    save it into DB
 +    save oldID => newID pair
 +
 +  for each element in (issues, notes)
 +    load data from DB
 +    search fields for links
 +    for each link found
 +      if link ID is known
 +        replace it with known newID
 +      else
 +        replace it with a conventional string or with URL to item in instance A
 +</code>
 +
 +===== Requirements =====
 +
 +
 +The code being developed depends on:
 +  * PHP 5
 +  * XMLReader module
 +  * XMLWRiter module
 +
 +
 +===== Other =====
 +
 +  * The resulting xml file can be tested for validity with xmllint like: <code>
 +xmllint --noout --dtdvalid mantis.dtd exported_issues.xml
 +</code>
  
-==== Implementation Log ==== +  * Check memory consumption when a huge number of issues are exported and imported 
- not started+  * Add checks for required extensions - bool extension_loaded  ( string $name  ) 
 +  let user choose destination project on instance B
  
-===== Other Changes ===== 
  
 +===== Chat Logs =====
  
 +<file>
 +Mar 16 22:27:57 vboctor giallu, I've submitted a note with my comments on the XML export.
 +Mar 16 22:28:30 nuclear_eclipse hi vboctor
 +Mar 16 22:28:48 vboctor howdy
 +Mar 16 22:35:13 giallu back
 +Mar 16 22:38:17 vboctor giallu, let me know your thoughts about the xml comments.
 +Mar 16 22:38:48 giallu have some time to discuss that now?
 +Mar 16 22:38:55 vboctor yes
 +Mar 16 22:39:17 giallu so, let's start from the DTD
 +Mar 16 22:39:43 nuclear_eclipse are you two talking about the DocBook manuals?
 +Mar 16 22:39:58 vboctor no, we are discussion an import/export xml format for Mantis.
 +Mar 16 22:40:07 nuclear_eclipse ah, ok, nm :P
 +Mar 16 22:40:16 giallu I just remembered seeing a dtd in bugzilla, so I read some documentation about DTD and adapted it to our purposes
 +Mar 16 22:40:36 giallu later I realized there exists XSD
 +Mar 16 22:41:11 vboctor ok, I guess this is not a big deal.  Once we define the format we can make a DTD, XSD or both.
 +Mar 16 22:41:13 giallu but I don't really want to complicate more this stuff (and I'm on a schedule becase as you probably remember this is a sponsored issue)
 +Mar 16 22:41:31 giallu then
 +Mar 16 22:41:59 giallu the DTD defines the fields _and_ their order
 +Mar 16 22:42:03 vboctor I don't mind if we don't implement all the specified requirements.  However, whatever we implement should fall into a preset vision.
 +Mar 16 22:42:04 thraxisp Are you writing import as well as export?
 +Mar 16 22:42:12 giallu thraxisp, yap
 +Mar 16 22:42:49 thraxisp A DTD will be a must for some one to use the import/export to interface to another system.
 +Mar 16 22:43:18 giallu thraxisp, see http://www.mantisbt.org/wiki/doku.php/mantisbt:importexport for more details
 +Mar 16 22:43:24 vboctor thraxisp, by DTD, you mean a DTD (or XSD) or specifically a DTD?
 +Mar 16 22:43:49 giallu I don't think it makes any difference if we have a DTD or XSD 
 +Mar 16 22:44:14 giallu but having a defined format is a prerequisite for writing conversion stuff
 +Mar 16 22:44:22 thraxisp As I recall, a DTD defines the structure and semantics on the XML file. (XSD may do the same).
 +Mar 16 22:45:08 giallu thraxisp, right. additionally AFAICT with XSD you have more powerful definition rules
 +Mar 16 22:45:31 giallu like this field contains strings, integers in this range, booleans etc
 +Mar 16 22:47:06 vboctor XSD and DTD both define schema.  However, XSD is in XML format and is a newer standard compared to DTD.
 +Mar 16 22:47:07 giallu vboctor, so, about your comment (2), I ordered the fields in the dtd with the same ordering I got them from filter_get_bug_rows
 +Mar 16 22:47:50 vboctor why do the two orders have to match?
 +Mar 16 22:48:03 giallu if we want a different ordering it's an additional step we need to carry somewhere, but
 +Mar 16 22:48:07 vboctor the ordering returned by filter_get_bug_rows can change any time.
 +Mar 16 22:49:08 vboctor Don't you get an array of issues, then you have your logic that translates the BugData into XML?  Hence, you have to specify the order anyway?
 +Mar 16 22:49:27 vboctor If you don't have this extra step, then a chance in the internal code will break the DTD right away.
 +Mar 16 22:49:32 giallu I wonder wht's the purpose of having a specific ordering, or having readable dates, etc. the file's purpose is meant to be a transport between two mantis instances
 +Mar 16 22:49:39 vboctor Also adding extra native fields will likely change your format as well.
 +Mar 16 22:50:06 vboctor Then why not use binary format?
 +Mar 16 22:50:09 vboctor or serialize?
 +Mar 16 22:50:29 vboctor The aim of using XML is to have a format that people can easily use to define their own tools based on a stable format.
 +Mar 16 22:50:35 giallu vboctor, anyway, yes, I can force an ordering on the fields, it's just that right now I haven't done it ;)
 +Mar 16 22:50:39 vboctor this can be export from product X and import into Mantis.
 +Mar 16 22:51:03 vboctor I think the schema shouldn't require a specific order.
 +Mar 16 22:51:19 vboctor there is no reason why field A must show up before field B.
 +Mar 16 22:51:30 giallu vboctor, the xml validates _only_ if the order is the same
 +Mar 16 22:51:52 giallu so whatever we write as order in the DTD, we need to use on the XML
 +Mar 16 22:52:16 giallu of course, that's just a validation issue
 +Mar 16 22:52:30 giallu and the data import would be perfectly fine with any ordering
 +Mar 16 22:53:37 vboctor In my opinion, the validation should ideally be a step that we execute before importing start.  This is to avoid failing mid way during import.  Specially given that we don't use transactions.  Unless we are planning to use transactions.
 +Mar 16 22:53:51 giallu so, for instance, I could ksort the array from filters and be done with this
 +Mar 16 22:54:34 vboctor I guess ordering is a not a big deal.
 +Mar 16 22:54:42 giallu yap. let's move on
 +Mar 16 22:55:05 vboctor I think the best option is to have a map which defines the name mapping / ordering, but it is not a big deal.
 +Mar 16 22:55:48 giallu so let's quicly summarize your comments:
 +Mar 16 22:55:56 giallu 1. The export has a lot of ids. These ids may mean different things to different people working with the data. I had the same problem when defining the SOAP API interface. What I ended up doing is to use an object reference concept. Hence, rather than just using an id, I use an id and a non-localized name to refer to an object. This applies to enumerations, users, etc.
 +Mar 16 22:56:42 thraxisp A map might also help for foreign import where fields are missing.
 +Mar 16 22:56:58 giallu I guess I can use the same system as used in SOAP. I'll add that as needed though. is that OK?
 +Mar 16 22:58:47 vboctor what do you mean by as needed?
 +Mar 16 22:59:06 * nt (n=tuckern@spc1-pete6-0-0-cust391.asfd.broadband.ntl.com) has joined #mantishelp
 +Mar 16 22:59:08 vboctor What I think we should do now is to have <reporter><id>1</id><name>administrator</name></reporter>
 +Mar 16 22:59:34 vboctor Same for view status: <view_state><id>10</id><name>public</name></view_state>
 +Mar 16 23:00:11 vboctor The other option is to use <view_state_id>10</view_state_id> and then have a section in the XML that defines all the possible view states.
 +Mar 16 23:00:38 giallu it means I could assume in a first instance that we are talking about two identical installations, so states are the same
 +Mar 16 23:01:05 vboctor But you can't assume that for users, projects, etc.
 +Mar 16 23:01:07 giallu then we can expand to your proposal in order to support some euristic (is that the word?)
 +Mar 16 23:01:50 vboctor how are you planning to handle users, projects, etc?
 +Mar 16 23:01:51 giallu vboctor, no. the users are to be handled like drafted in the reqiurement page 
 +Mar 16 23:01:56 giallu i.e.
 +Mar 16 23:02:49 * vboctor checked the wiki page.
 +Mar 16 23:02:53 giallu define a squash-to userid during import and ask admin if everything has to be squashed to that user or just the not matching ids
 +Mar 16 23:02:58 vboctor your sample xml didn't include user names.
 +Mar 16 23:03:42 giallu for projects. I think I'm limiting the import to a _single_ project
 +Mar 16 23:04:12 giallu I don't really want to guess which project an issue belongs to
 +Mar 16 23:04:34 giallu vboctor, which names?
 +Mar 16 23:04:58 giallu reporters?
 +Mar 16 23:05:14 vboctor I just think having <project><id>5</id><name>Mantis</name></project> will allow us to match by name even if ids don't match.
 +Mar 16 23:05:29 vboctor by user names, i meant reporters, handlers, etc.
 +Mar 16 23:05:50 giallu so basically, everywhere we have an id...
 +Mar 16 23:06:08 CIA-11 mantisbt: thraxisp * r5121 /branches/BRANCH_1_1_0/mantisbt/adm_config_report.php:
 +Mar 16 23:06:08 CIA-11 mantisbt: partial fix for #0008976: Remote Code Execution in adm_config
 +Mar 16 23:06:08 CIA-11 mantisbt:  - hide update form for those who can't change items
 +Mar 16 23:08:36 vboctor giallu, yes.  I think it will make the format more flexible and we won't have to make breaking changes.
 +Mar 16 23:09:24 giallu vboctor, well you know, the X in XML means we shouldn't have this problem ;)
 +Mar 16 23:10:38 vboctor just end up with more code to maintain for no strong reason.
 +Mar 16 23:11:02 vboctor now we need to handle <reporter_id> and <reporter>
 +Mar 16 23:11:34 vboctor do you think it is a lot more work to export id/name combinations?
 +Mar 16 23:11:58 vboctor or is the logic to be intelligent about such combinations on import is what is worrying you?
 +Mar 16 23:12:36 giallu yeah. it's just more work in the import side (which is still a bit foggy in my head)
 +Mar 16 23:12:57 giallu I think I can do that
 +Mar 16 23:13:06 giallu just one more Q
 +Mar 16 23:13:33 giallu do you think your 4 years old note about using DOM XML still apply?
 +Mar 16 23:14:02 giallu if possible, I'd rather use some existing php module for the job
 +Mar 16 23:14:27 giallu though right now I'm writing it with custom code
 +Mar 16 23:14:35 CIA-11 mantisbt: thraxisp * r5122 /trunk/mantisbt/adm_config_report.php:
 +Mar 16 23:14:35 CIA-11 mantisbt: partial fix for #0008980: Port: Remote Code Execution in adm_config
 +Mar 16 23:14:35 CIA-11 mantisbt:  - hide change form from unauthorized users
 +Mar 16 23:15:17 vboctor custom code is fine.
 +Mar 16 23:16:11 giallu yes. but I'm asking if it's fine to use some xml related functionality. The parsing is trickier
 +Mar 16 23:17:50 vboctor I don't understand your question, probably since I don't remember the comment you are referring to.
 +Mar 16 23:17:58 * vboctor checking...
 +Mar 16 23:18:13 giallu ah. you asked to _not_ use DOM XML
 +Mar 16 23:18:34 giallu becasue it was possibly not available everywhere
 +Mar 16 23:19:42 vboctor yep, just read it now.
 +Mar 16 23:19:48 vboctor what is the status of DOM XML and PHP 5?
 +Mar 16 23:19:59 vboctor I remeber there was also Simple XML in PHP 5?
 +Mar 16 23:20:36 giallu right
 +Mar 16 23:21:02 vboctor Given that export / import is not a core feature, I am ok with using a PHP extension.
 +Mar 16 23:21:09 giallu great
 +Mar 16 23:21:17 vboctor Similar to twitter notifications that is dependent on curl extension.
 +Mar 16 23:21:37 giallu I'm probably leaving the exporter as is unless I find it unmaintenable
 +Mar 16 23:22:21 giallu but I'm definitely looking for some cooked solution about import
 +Mar 16 23:22:43 vboctor consistency between export / import would be ideal.
 +Mar 16 23:23:06 vboctor you also need to be careful that PHP doesn't run out of memory when exporting / importing a lot of issues.
 +Mar 16 23:23:47 giallu ok. I'm noting this on the wiki page
 +Mar 16 23:33:20 vboctor good
 +</file>
mantisbt/importexport.1204382769.txt.gz · Last modified: 2008/10/29 04:31 (external edit)

Driven by DokuWiki