通过CLI的导入数据

目的

iTop根据源csv文件在itop中提供了通用Web服务到导入数据。该服务可以通过命令行界面在服务上本地使用,也可以通过HTTP发布请求远程使用。

使用此服务,您可以通过批处理脚本轻松地在iTop中使用导入数据,该批处理脚本从第三方工具提取数据并生成csv格式的文件。例如,脚本数据从您的Active Directory中提取了数据的列表。

当涉及到导入的大量数据时,必须使用CLI模式,因为您将无法达到Web服务器的限制(超时和内存限制)。

与交互式CSV导入相反,CLI导入允许您指定任何属性来协调外部密钥。缺少某些对帐键时,这是变通方案。示例:软件类仅具有名称作为对帐键(数据模式下的默认设置)。您需要导入DBServer对象及其各自的软件信息(指向软件导入的外部密钥)。只要同一软件具有两个版本(名称相同,含义不明确!),交互式CSV导入就会发出错误。在CLI导入中,您可以同时指定Software-> Name和Software->对象和导入歧义性问题。

用法

命令行界面

命令行的所有参数在下面的“参数”部分中列出。下一行是必需参数的示例。所有参数均以-开头

php /var/www/itop/webservices/import.php --auth_user=login --auth_pwd=password --class=Organization --csvfile=file --reconciliationkeys="list of keys"

协调键对于唯一标识要在iTop中导入的元素非常重要。协调密钥必须存在于数据文件中。与交互式CSV导入相比,您不限于默认情况下为该类定义的对帐键列表。

当可以在外键上执行对帐时,可以指定任何目标类属性。示例:您正在导入Person对象,并且需要在位置上进行协调。您可以将Location-> postal_code指定为对帐键之一(奇怪但…)。

此外,外部密钥将由对帐方案中指定的所有属性唯一标识。在我们的示例中,如果您认为NameeCountry唯一地标识位置,则在对帐方案中同时使用Location-> Name和Location-> Country。

争论

论据描述Defaut价值
param_file参数文件-请参阅参数文件-
auth_user用户登录名-仅CLI模式-
auth_pwd用户密码-仅CLI模式-
class加载对象的类别必须!
csvdataCSV内容(仅HTTP模式,数据可以通过此页面参数发布)HTTP模式下必选
csvfile源文件的路径和名称强制在CLI模式下
charsetCSV数据的字符集编码:UTF-8、ISO-8859-1、WINDOWS-1251、WINDOWS-1252、ISO-8859-15使用iTop配置文件中的csv_file_default_charset,默认值为'ISO-8859-1'
date_format输入日期格式(用于日期和日期时间)-示例:Y-m-d,ddmmY(法国)<空白>
separatorCSV数据中的列分隔符,(昏迷)
qualifierCSV数据中的限定符”(双引号)
outputretcode返回错误中的行的计数,“总结”返回简明的报告,“详细信息”以获取详细的报告(列出的每一行)总结
reconciliationkey列的标签或代码(取决于no_localize标志)(以逗号分隔),用于标识现有对象并更新它们或创建一个新对象<空白>
simulate如果设置为1,则将不执行加载,但是将产生预期的报告0
with_archive如果设置为1,则导入将检索已归档的对象作为对象进行更新并作为有效的外部密钥0
comment要添加到变更日志中的注释<空白>
no_localize如果设置为0,则应以登录的用户的语言本地化列标题,值和对帐键。设置为1以使用内部属性代码和值(枚举)0 =标签不区分大小写

每个对帐密钥必须存在于源文件头中,并以相同的方式写入

输出

命令行将返回总结报告(请参见下文),总结用于导入的参数和导入的结果:

  • 创建的元素数
  • 更新的元素数
  • 错误中的元素数量

如果您指定了选项–result = details,则对于使用导入的状况导入的每个元素,将获得一行(请参见下面的示例)

结果

#Output format: details
#Class: Organization
#Separator: ;
#Qualifier: \"
#Charset Encoding:UTF-8
#Data Size: 26
#Data Lines: 1
#Simulate: 0
#Columns: name, code
#Reconciliation Keys: name
#Change tracking comment: 
#Issues: 0
#Warnings: 0
#Created: 1
#Updated: 0
#Unchanged: 0
Line;Status;Object Class;Object Id;name (Name);code (Code)
0;created;Organization;3;\"WorldCompany\";\"WCY\"

可能的错误

遗失论点

ERROR: Missing argument 'class'

当缺少必需参数时,将显示此类错误

错误的论点

#Unknown class: 'toto'

如果指定的类在iTop中不存在,则会发生此错误。请注意,类名区分大小写。

#Unknown reconciliationkeys: 'attribute'

如果对帐密钥列表中的至少一列未知,则会发生此错误。
原因可能是:

  • 提供了代码而不是标签,反之亦然
  • 对帐键是“外部字段”
  • CSV文件中不存在对帐键

不能将外部字段(例如org_name)用作对帐键,请改用org_id-> name

#Unknown column: 'Last Nme'

如果csv文件中的列未知,则会发生此错误

当使用默认值no_localize = 0时,必须在CSV文件标题和对帐键中使用标签。出于历史原因,在CLI模式下,标签不区分大小写,因此reconciliationkeys =“ name,Owner organization-(即标准)均有效。

允许的用户

始终将允许管理员执行此页面。

如果允许非管理员修改给定类的对象,则将允许它们。

导入图像和文件

某些类将图像或文件文档作为其字段之一。例如,Person类包含一个Picture字段,其中包含此人的图片。以交互方式编辑人物时,用户可以从其计算机上载图像以提供人物图片。但是,在执行CSV导入时,由于CSV格式不支持此文件,因此无法“上传”此类文件或将其内容放入CSV文件本身。

在这种情况下,CSV文件必须在相应的CSV列中提供一个URL,以便从中上传文件。必须从iTop服务器可以访问此URL,并返回所需的图像(或文件文档)。如果上载失败或上载文档的格式与该字段的预期格式不匹配(例如,URL不返回的图像),则import/update对于该字段(以及整行) CSV将被拒绝)。

局限性:

URL必须直接指向要上传的图像(不支持重定向)

必须从iTop Web服务器访问URL(请注意,某些公众网站可能拒绝访问PHP脚本-它们会检查HTTP用户处理人员字符串)

URL不需要身份验证,因为上传是由iTop服务器本身完成的

支持指向iTop本身的URL(例如CSV/Excel导出提供的URL)。但是仅当它们指向同一个iTop实例时(在这种情况下导入才会解析提供的URL并从iTop数据库中读取文档)。

如果您使用管理员账号执行CSV导入,则可以指定iTop服务器本身上的文件的路径(但这仅适用于管理员)。

用例:将带有相应登录名的人员导入iTop

此用例描述了如何在iTop中对导入人员及其登录名进行操作。这是从Active Directory加载这些信息的一种方法。在这种情况下,您只需要编写一个脚本即可从您的Active Directory创建两个csv文件person.csv和登录名。csv。

进口人

我们想创建两个联系人:

  • 属于组织Demo的Claude Monet
  • 组织IT部门的Gustave Flaubert

在此示例中,这两个组织已经存在于iTop中

导入的类是UserLocal,csv文件(登录名。csv)中的数据是:

人.csv

Last Name,Status,Organization->Name,Email,Phone,Notification,Function,First Name
"Monet","Active","Demo","monet@demo.com","","yes","","Claude"
"Flaubert","Active","IT Department","flaubert@it.com","","yes","","Gustave"

然后运行以下命令行:

php webservices/import.php --auth_user=admin --auth_pwd=admin --csvfile="person.csv"
 --class="Person" --reconciliationkeys="Email"

导入用户帐户

我们想创建两个用户账号:

  • 支持与联系人相关的配置经理和变更管理员flaubert@it.com
  • 门户是与联系人相关的porta用户monet@demo.com

在此示例中,登录名通过发送邮件链接到联系人,我们导入简档(角色)链接到联系人。查看用于导入联系人的特定语法。有关导入链接集的更多信息,请参阅导入链接集

导入的类是UserLocal:

登录名.csv

Contact (person)->Email,Login,Language,Password,Profiles
"flaubert@it.com","support","EN US",1234,profileid->name:Configuration Manager|profileid->name:Change Supervisor
"monet@demo.com","portal","EN US",1234,profileid->name:Portal user

然后运行以下命令行:

php webservices/import.php --auth_user=login--auth_pwd=password --csvfile="login.csv"
  --class="UserLocal" --reconcialiationkeys="Login"

汇入位置

位置.csv

然后运行以下命令行:

"name","org_id->name","country","city"
"Bordeaux","Demo","France","Bordeaux"
"Grenoble","Demo","France","Eybens"
"Paris","Demo","France",""

then run the following command line:

php webservices/import.php --auth_user=login --auth_pwd=password --csvfile="location.csv"
   --charset="UTF-8" --no_localize=1 --class="Location" --reconcialiationkeys="name,org_id->name"

局限性已知问题

外部字段(例如virtualhost_id-> org_name)不能用作对帐键。

变通方案:org_name可以替换为org_id-> name

局限性:箭头运算符(->)不能多次使用(即不支持virtualhost_id-> org_id-> name)。

此导入机制无法抑制iTop对象,请使用数据同步为了这

历史

0.9:该特性的第一个实施

2.0:添加了标记no_localize

原贴链接:https://www.itophub.io/wiki/page?id=2_7_0%3Aadvancedtopics%3Aimportcli


Import data via CLI

Purpose

iTop provides a generic web service to import data in itop based on a source csv file. This service can be used locally on the server via the command line interface or remotely via an HTTP post request.

Using this service, you can easily import data in iTop via a batch script that extract data from a third party tool and generate a csv formatted file. For instance a script data extract a list of contact from your Active Directory.

When it comes to import a real huge amount of data, the CLI mode will be a must because you will not reach the limitations of the web server (timeout and memory limits).

As opposed to the interactive CSV import, the CLI import lets you specify any attribute to reconcile the external keys. This is a workaround when some reconciliation keys are missing. Example: the class Software has only the name as reconciliation key (default setting in the datamode). You need to import DBServer objects with their respective software information (an external key pointing to the software object). The interactive CSV import will issue errors whenever the same software has two versions (same name, ambiguous!). In the CLI import you can specify both Software->Name and Software->Version and resolve the ambiguity issue.

Usage

Command line interface

All the parameters of the command line are listed in the section “Arguments” here after. The following line is an example of the mandatory argument. All arguments starts with --

php /var/www/itop/webservices/import.php --auth_user=login --auth_pwd=password --class=Organization --csvfile=file --reconciliationkeys="list of keys"

The reconciliation keys are very important to identify uniquely the element to be imported in iTop. The reconciliation keys must be present in the data file. As opposed to the interactive CSV import, you are not limited to the list of reconciliation keys defined by default for the class.

When the reconciliation can be performed on an external key, you can specify any of the target class attributes. Example: you are importing Person objects and need to reconcile on the location. You can specify Location->postal_code as one of the reconciliation keys (weird but…).

Moreover, the external key will be uniquely identified by all its attributes specified in the reconciliation scheme. In our example, if you consider that Name/Country uniquely identifies the locations, the use both Location->Name and Location->Country in your reconciliation scheme.

Arguments

ArgumentDescriptionDefaut value
param_fileParameters file - see Parameters file-
auth_userUser login - CLI mode only-
auth_pwdUser password - CLI mode only-
classClass of the loaded objectsMandatory!
csvdataCSV contents (HTTP mode only, data may be posted through this page argument)Mandatory in HTTP mode
csvfilePath and name of the source fileMandatory in CLI mode
charsetCharacter set encoding of the CSV data: UTF-8, ISO-8859-1, WINDOWS-1251, WINDOWS-1252, ISO-8859-15use csv_file_default_charset from iTop configuration file, which default is 'ISO-8859-1'
date_formatInput date format (used both for dates and datetimes) - Examples: Y-m-d, d/m/Y (France)<blank>
separatorColumn separator in CSV data, (coma)
qualifierQualifier in CSV data“ (double quote)
outputretcode to return the count of lines in error, “summary” to return a concise report, “details” to get a detailed report (each line listed)summary
reconciliationkeyslabel or code (depends on no_localize flag) of the columns (comma separated) used to identify existing objects and update them, or create a new one<blank>
simulateIf set to 1, then the load will not be executed, but the expected report will be produced0
with_archiveIf set to 1, then the import will retrieve archived objects as object to update and as valid external key0
commentComment to be added into the change log<blank>
no_localizeIf set to 0, then column headers, values and reconciliationkeys are supposed to be localized in the language of the logged in user. Set to 1 to use internal attribute codes and values (enums)0 = label not case sensitive

Each reconciliation key must be present in the source file header and written the same way

Output

The commande line will return a summary report (see below) summarizing the argument used for the import and the result of the import:

  • Number of elements created

  • Number of elements updated

  • Number of elements in error

In case you have specified the option –result=details you will get one line for each element imported with the status of the import (see example below)

Result

#Output format: details
#Class: Organization
#Separator: ;
#Qualifier: \"
#Charset Encoding:UTF-8
#Data Size: 26
#Data Lines: 1
#Simulate: 0
#Columns: name, code
#Reconciliation Keys: name
#Change tracking comment: 
#Issues: 0
#Warnings: 0
#Created: 1
#Updated: 0
#Unchanged: 0
Line;Status;Object Class;Object Id;name (Name);code (Code)
0;created;Organization;3;\"WorldCompany\";\"WCY\"

Possible errors

Missing argument

ERROR: Missing argument 'class'

Such error is displayed when a mandatory argument is missing

Wrong argument

#Unknown class: 'toto'

This error occurs if the class specified does not exists in iTop. Be careful the class name is case sensitive.

#Unknown reconciliationkeys: 'attribute'

This error occurs if at least one column in the reconciliation keys list is not known.
The reason can be:

  • a code instead of a label was provided or vice-versa

  • the reconcialiation key is a External Field

  • the reconcialiation key is not present in the csv file

An External Field such as org_name cannot be used as a reconciliation key, use instead org_id->name

#Unknown column: 'Last Nme'

This error occurs if a column is unknown in the csv file

When using no_localize=0 which is the default, you must use the labels in the CSV file headers and in reconciliationkeys. For historical reason, in CLI mode, labels are not case sensitive soreconciliationkeys=“name,owner organization->name” works as well as =“Name,Owner organization->Name” which would be the standard.

Allowed users

Administrators will always be allowed to execute this page.

Non administrators will be allowed if they are allowed to modify objects of the given class.

Importing Images and Files

Some classes contain an image or a file document as one of their field. For example, the Person class, contains a Picture field which holds the picture of this person. When editing a Person interactively, the user can upload an image from her computer to provide the picture of the person. However when performing a CSV import, it is not possible to “upload” such a file or to put its content inside the CSV file itself, since the CSV format does not support this.

In such a case the CSV file must provide - in the appropriate CSV column - an URL to upload the file from. This URL must be accessible from the iTop server and return the expected image (or file document). If the upload fails or if the format of the uploaded document does not match the expected format for the field (for example if the URL does not return an image for the picture field), the import/update will fail for this field (and the whole line of the CSV will be rejected).

Limitations:

  • The URL must point directly to the image to upload (redirections are not supported)

  • The URL must be accessible from the iTop web server (beware some public websites may deny access to a PHP script - they check the HTTP User Agent string)

  • The URL must not require authentication, since the upload is done by the iTop server itself

  • URLs pointing to iTop itself (like the URLs provided by CSV/Excel export) are supported. But only if they point to the same iTop instance (in such a case the import will parse the provided URL and will read the document from the iTop database).

  • If you use an administrator account to perform the CSV import, you can specify a path to a file located on the iTop server itself (but this works only for administrators).

Use case: importing persons with their corresponding login into iTop

This use case describes how to import persons and their login in iTop. This is one way to load those information from an Active Directory. In that case you just have to write a script that create the two csv files person.csv and login.csv from your Active Directory.

Importing persons

We would like to create two contacts:

  • Claude Monet who belongs to organization Demo

  • Gustave Flaubert who belongs to organization IT Department

In this example the two organizations already exist in iTop

The class to import is UserLocal, and the data in csv file (login.csv) are:

person.csv
 
Last Name,Status,Organization->Name,Email,Phone,Notification,Function,First Name
"Monet","Active","Demo","monet@demo.com","","yes","","Claude"
"Flaubert","Active","IT Department","flaubert@it.com","","yes","","Gustave"

then run the following command line:

php webservices/import.php --auth_user=admin --auth_pwd=admin --csvfile="person.csv"
 --class="Person" --reconciliationkeys="Email"

Importing user accounts

We would like to create two user account:

  • support is configuration manager and change supervisor, related to the contact flaubert@it.com

  • portal is a Porta user related to the contact monet@demo.com

In this example the login are linked to a contact via the email and we import the Profiles that are linked to the contact. Look at the particular syntax for importing the profiles. For more information about importing link sets refer to Import a LinkedSet

The class to import is UserLocal:

login.csv
 
Contact (person)->Email,Login,Language,Password,Profiles
"flaubert@it.com","support","EN US",1234,profileid->name:Configuration Manager|profileid->name:Change Supervisor
"monet@demo.com","portal","EN US",1234,profileid->name:Portal user

then run the following command line:

php webservices/import.php --auth_user=login--auth_pwd=password --csvfile="login.csv"
  --class="UserLocal" --reconcialiationkeys="Login"

Importing Locations

location.csv
 
"name","org_id->name","country","city"
"Bordeaux","Demo","France","Bordeaux"
"Grenoble","Demo","France","Eybens"
"Paris","Demo","France",""

then run the following command line:

php webservices/import.php --auth_user=login --auth_pwd=password --csvfile="location.csv"
   --charset="UTF-8" --no_localize=1 --class="Location" --reconcialiationkeys="name,org_id->name"

Limitations / Known issues

  • External Fields (such as such as virtualhost_id->org_name) cannot be used as reconciliation keys.

  • Workaround: org_name can be replaced by org_id->name

  • Limitation: the arrow operator (->) cannot be used mutiple times (i.e. virtualhost_id->org_id->name is not supported).

This import mechanism cannot suppress iTop object, use DataSynchro for this

History

  • 0.9: first implementation of this feature

  • 2.0: added the flag no_localize

标签:
由 superadmin 在 2020/08/27, 16:21 创建
    

需要帮助?

如果您需要有关XWiki的帮助,可以联系:

深圳市艾拓先锋企业管理咨询有限公司