数据同步参考

如果您不熟悉此概念,请从总览

数据来源定义

据源使用菜单“管理工具同步数据源”定义。

Admin Tools/Synchronization Data Sources

每个数据源定义:

  • 要同步的对象的目标类
  • 要同步的对象的属性/字段列表
  • 如何搜索/协调对象与 iTop 中已有的对象
  • 在iTop中同步更新和删除对象的规则
  • iTop终端用户的同步对象的行为方式(哪些字段是只读的,允许用户删除对象…)
  • 可选的超链接和图标,用于将iTop用户引至外部应用中的相应对象

Definition of a synchro data source

描述

属性的这一部分为此同步数据源定义了一些常规参数。

属性强制性的吗?含义样品价值
名称强制性的引用此数据来源的名称来自SCCS的服务器同步
描述可选的附加说明,仅供参考 
状况强制性的实施,已生产或已淘汰:用于过滤和分类数据来源生产
目标阶层强制性的动态的对象类服务器
用户强制性的用户允许执行此同步管理员
联系人通知可选的联系人,用于在执行同步时(通过发送邮件)通知结果,并包含至少一个错误 
图标的超链接可选的表示此数据源的图像的超链接(URL)。在iTop中同步的图标标题旁边的“锁定”符号上显示的工具提示中,显示此图标而不是默认图标。 
应用的超链接可选的同步到数据的远程应用的超链接(URL)。超链接可以包含占位符,例如$ this-> attcode $(将由当前对象的属性的价值替换)或$ replica-> column_name $(将由占位符中的命名列的价值替换)。复制品)https:::myapp.acme.commdisplayyserverr$replica->primary_key$
数据表可选的将在其中存储副本信息的MySQL表的名称。如果省略,系统将为您生成一个唯一的名称。synchro_data_servers_from_sccs

将基于类数据模型创建用于保存副本的表,如果稍后更改数据模型,则该表将通过安装程序进行更新(与往常一样,将添加但不删除列)。
因为需要通过同步导入 为空字段(例如日期),发布2.3.4中的日期列格式已从date更改为varchar(10)

搜索和对帐

此部分属性定义

  1. 副本表中包含的信息如何与iTop数据库的内容匹配
  2. 同步数据源有关创建新iTop对象的行为是什么。
属性含义
对帐策略使用属性:创建新副本时,系统将基于同步属性列表(将Reconciliation标记设置为Yes),在iTop中将搜索匹配对象。
使用primary_key:副本的primary_key列应包含iTop中对象的标识符。
功能归零对帐未在iTop中找到任何对象时该怎么办。
Create指示系统创建目标类的新对象。
在这种情况下,错误将导致系统至报告作为错误(即,该同步数据源应该仅更新现有对象)。
功能上一个对帐在iTop中找到一个完全匹配的对象时该怎么办。
更新指示系统更新此对象。
在这种情况下,错误将导致系统至报告作为错误(即,该同步数据源应该创建对象,而从不更新现有对象)。
功能上的很多对帐在iTop中找到多个匹配的对象时该怎么办。
Create指示系统创建另一个对象。
错误将导致系统到报告成为错误。
拿第一个指示系统选取列表中的第一个对象并进行更新。

删除规则

本节定义如何处理同步对象的删除:

  1. 通过iTop中的终端用户,
  2. 通过同步系统本身。
属性含义
允许的用户哪些终端用户允许删除iTop中的同步对象。
同步对象时,没有人不允许在iTop中删除它。只有同步系统才能删除它。
管理员仅允许管理员删除同步的对象。
每个人都允许删除此类对象,正常访问规则适用于此类同步对象。
满载间隔同步的两次执行之间的持续时间,被视为已加载(接触)所有副本。如果未触摸副本的时间超过此价值的持续时间,则认为该副本已过时。
删除策略复制副本过时该怎么办。
忽略任何操作,关联的对象仍保持在iTop中。
删除删除iTop中的关联对象(以及数据表中的副本)。
更新根据更新规则(请参阅下文)指定,更新关联的对象。
更新然后删除按照更新规则(请参阅下文)指定的方法更新关联的对象,然后在“保留期限”之后删除对象(和副本)。
更新规则attribute_code:value 值的用分号分隔的值列表,用于指定要设置关联对象的哪个属性以及将哪个价值设置。示例:价status:obsolete;description:no longer synchronized.
保留期限当“删除策略”设置为“更新然后删除”时,关联的对象更新与删除之间的持续时间。

如果在运行同步_exec之前在同一个数据源上运行多个同步_导入,请不要使Full load interval = 0s保留,否则删除同步将不起作用。

参数的满载间隔将被忽略synchro_import –synchronize=1,因为该唯一导入中未提供的任何对象被视为“过时”

如果您将满负载间隔设置为7天,保留期限为14天,则3周后将删除源不再提供的对象

属性

Synchro Data Source Attributes

属性标签

“属性”标签定义:

  1. 如何确定是必须在iTop中创建新的对象还是必须将现有的对象与副本相关联(如果调节规则在属性中为“使用属性”)
  2. 关联的对象的哪些属性必须更新,以及如何更新

当synchro_data_xxx 表中出现新的记录时,同步机制将尝试将此记录与现有的iTop对象关联。如果一个或多个属性在“对帐?”中标记为“是”。列,这些列属性将用于与所有这些值匹配的iTop对象的记录。如果找到一个对象,它将与该记录关联。如果找不到对象,将创建一个新的iTop对象。

适当对帐规则的定义是数据同步正确行为的关键。为了使同步正常进行,必须使用属性列表作为对帐规则,该属性以可靠的方式唯一地标识动态的对象。

更新列

更新列定义是否必须更新属性。

  • Checked = Yes:将更新属性
  • Unchecked = No:此属性的外部源信息将被忽略,iTop对象上的现有价值将保持不变。
更新策略列

更新策略列定义了更新的属性的行为。可能的值为:

  • 锁定:属性将完全由数据同步驱动,并在iTop用户界面中显示为只读(旁边有一个小锁图标),
  • 解锁:属性将通过同步进行更新(无论源数据何时更改),但终端用户仍然可以在iTop中并行修改属性,
  • 如果为空,则初始化:如果iTop中的价值为空,则同步只会设置该属性的价值。属性仍可在iTop内部进行修改。
重新对帐键列

最后,“对帐键”列定义了如何流程属性,这些属性是对另一个iTop对象的引用(即外键)。提供的数据可以包含外键(id)的实际价值,也可以包含另一个属性,例如对象的名称。它必须唯一地标识对象。

对于外键,iTop不支持对多个属性(例如名称AND组织)的组合进行对帐。如果出现这种需求,那么在将ID提供给iTop之前,必须使用ETL或脚本执行所需的对帐。

状况

状况选项卡

状况选项卡显示有关同步执行的信息。如果从未运行过同步,则该选项卡显示以下消息:

该同步从未运行过。还没有日志。

否则,该选项卡将显示类似于以下模式的模式:

Synchro Data Source Status

同步历史

在屏幕的左侧,同步的每次执行都显示在历史列表中,显示了整个状况,执行的开始日期和时间。

单击此列表中的元素,以显示相应的状况详细信息,显示在右侧的两列中。

左侧的列显示 synchro_data_xxxx表中的记录的状况:

  • 新记录是自上一次执行同步以来已添加到表中的记录,
  • 现有记录是自从上一次执行同步以来已经存在并重新加载的记录,
  • 消失的记录是由于超过full_load_interval持续时间而没有加载的记录。
  • 忽略的记录对应于iTop中不再存在的对象。

右边的列根据对iTop中相关对象的修改显示了同步的后果。

  • 对于每个新记录,后果可能是:
    • 在iTop中创建了新的对象(Created),
    • iTop中已经存在的对象的更新(Updated),
    • 与iTop中的对象的简单关联,它已经是最新的(Unchanged),
    • 或尝试在iTop中创建或更新对象时使用错误(最可能是由于数据不一致)。
  • 对于每个现有记录,后果可能是:
    • 关联的对象的更新(Updated),
    • 完全没有变更(Unchanged)
    • 或尝试更新关联的对象时出现错误(Errors)
  • 对于每个消失的记录,后果(“删除规则”上的依赖)可以是:
    • iTop中关联的对象已被删除(Deleted),
    • iTop中关联的对象已更新(Updated),
    • iTop中关联的对象保持不变(No Action),
    • 或尝试更新或删除iTop中关联的对象时发生错误(Errors)。

复制品

在iTop中,副本用于存储状况信息以及synchro_data_xxx表中加载的原始数据与关联的iTop数据之间的关系。

通过单击状况选项卡顶部的链接,可以显示给定数据同步源的副本。副本显示如下:

Synchro Replica

副本的内容对于了解数据同步机制的行为(并获取有关错误的更多信息)很有用,但是其数据由同步维护,并且永远不要手动修改。

使用MySQL语句填充synchro_data_xxxx

如果使用SQL命令(使用ETL或自制脚本)填充表synchro_data_xxx,则必须注意不要创建重复的记录。请记住,synchro_data_xxx表中的每个记录都对应于iTop中的一个同步。

为避免此类重复,必须先将查询用于现有的记录,然后再将新的记录插入表中。为了加快此查询的运行速度,您可以构建一个唯一标识“源”对象的查询并将该查询存储在表的“ primary_key”列(具有唯一的价值)中,或者使用记录列的任何组合并更改记录_价值_xxxx表的定义以添加您自己的索引。

要使用MySQL申明INSERT…ON DUPLICATE KEY UPDATE在synchro_data_xxx表中插入记录。依靠“插入前”触发器的内部iTop机制不支持此功能。

如果您需要清空synchro_data_xxxx表,请不要使用MySQL申明TRUNCATE TABLE,因为该申明实际上将删除并重建该表……但不会触发它!而是使用synchro_data_xxxx的DELETE。

然后触发器通过执行页面同步在iTop对象和临时表的副本之间执行“synchro/synchro_exec.php”(无论是在命令行模式下还是通过调用网页(例如使用wget))。

使用synchro_import填充synchro_data_xxxx 

看到:synchro_import.php

动态参数的synchro_import.php 触发器 synchro_exec.php 默认

如何指定相关对象的列表(链接集)

一些类具有称为链接集的属性。示例:UserLocal.profile_list

您可以通过在相应的列中写入所有信息来动态这些列表:

这是一个示例,指定登录名将具有一个简档:'门户用户'

INSERT login, profile_list INTO data_synchro_userlocal_1 VALUES ('johndoe', 'profileid->name:Portal User;reason:Customer')...;

您还可以指定几个链接(几个角色)。

更多信息导入链接集

原贴链接:https://www.itophub.io/wiki/page?id=2_7_0%3Aadvancedtopics%3Adata_synchronization


Data Synchronization Reference

If you are not familiar with the concept, start by the Overview

Data Source Definition

The Data Sources are defined using the menu Admin Tools/Synchronization Data Sources.

Admin Tools/Synchronization Data Sources

Each data source defines:

  • The target class of the objects to be synchronized

  • The list of attributes/fields of the objects to synchronize

  • How to search/reconcile the objects with the objects already existing in iTop

  • The rules of synchronizing/updating and possibly deleting objects in iTop

  • How the synchronized objects behave for the iTop end-users (which fields are read-only, are users allowed to delete the objects…)

  • An optional hyperlink and icon to refer iTop users to the corresponding object in the external application

Definition of a synchro data source

Description

This section of the Properties defines some general parameters for this Synchro Data Source

PropertyMandatory?MeaningSample value
NameMandatoryThe name which refers to this data sourceServers synchro from SCCS
DescriptionOptionalAn additional description, for information only 
StatusMandatoryimplementation, production or obsolete: for filtering/classifying the data sourcesproduction
Target ClassMandatoryThe class of objects to synchronizeServer
UserMandatoryThe user allowed to execute this synchroadmin
Contact to notifyOptionalThe contact to notify (by email) of the results when the synchronization is executed and contains at least one error 
Icon's hyperlinkOptionalThe hyperlink (URL) to an image representing this data source. This icon is shown instead of the default icon in the tooltip appearing on the “Lock” symbol next to the title of the synchronized object in iTop 
Application's hyperlinkOptionalThe hyperlink (URL) to the remote application from which the synchronized data comes from. The hyperlink can contain placeholders such as $this->attcode$ (which will be replaced by the value of the attribute of the current object) or $replica->column_name$ (which will be replaced by the value of the named column from the replica)https://myapp.acme.com/display/server/$replica->primary_key$
Data tableOptionalThe name of the MySQL table in which the replica information will be stored. If omitted, the system will generate a unique name for you.synchro_data_servers_from_sccs

The table which will hold the replicas, is created based on the class datamodel, if the datamodel is changed later this table will be updated by the setup (as usual column will be added but not removed).
Because it was required to be able through synchro import to empty fields such as date, the format of date column was changed in release 2.3.4 from date to varchar(10)

Search & reconciliation

This section of the properties defines

  1. How the information contained in the replica table is matched against the content of the iTop database

  2. What is the behavior of this synchro data source concerning the creation of new iTop objects.

PropertyMeaning
Reconciliation policyUse the attributes: when a new replica is created, the system will search for a matching object in iTop based on the list of Synchro attributeswith the flag Reconciliation set to Yes.
Use primary_key: the column primary_key of the replica is expected to contain the identifier of the object in iTop.
Action on zeroWhat to do when the reconciliation did not find any object in iTop.
Create instructs the system to create a new object of the Target Class.
Error will cause the system to report this case as an error (i.e. this synchro data source is supposed to only update existing objects).
Action on oneWhat to do when the reconciliation found exactly one matching object in iTop.
Update instructs the system to update this object.
Error will cause the system to report this case as an error (i.e. this synchro data source is supposed to create objects and never update existing objects).
Action on manyWhat to do when the reconciliation found several matching objects in iTop.
Create instructs the system to create another object.
Error will cause the system to report this case as an error.
Take the first one instructs the system to pick the first object in the list and to update it.

Deletion rules

This section defines how the deletion of the synchronized objects are handled:

  1. by the end-users in iTop,

  2. by the synchronization system itself.

PropertyMeaning
Users AllowedWhich end-users are allowed to delete synchronized objects in iTop.
Nobody when an object is synchronized, nobody is allowed to delete it in iTop. only the synchronization system can delete it.
Administrators only only administrators are allowed to delete a synchronized object.
Everybody allowed to delete such objects the normal access rules apply to such synchronized objects.
Full load intervalThe duration between two executions of the synchronization which are considered as having loaded (touched) all replicas. When a replica has not been touched for a duration greater than this value, it is considered as obsolete.
Delete PolicyWhat to do when a replica becomes obsolete.
Ignore do nothing, the associated object remains as is in iTop.
Delete Delete the associated object in iTop (and the replica in the data table).
Update Update the associated object as specified by the Update rules (see below).
Update then Delete Update the associated object as specified by the Update rules (see below), then after the Retention Duration, delete the object (and the replica).
Update rulesA list of values separated by semi-colons, of the form attribute_code:value to specify which attribute of the associated object to set and to which value. Example: status:obsolete;description:no longer synchronized.
Retention durationThe duration between the update of the associated object and its deletion when the Delete Policy is set to Update then delete.

If you run multiple synchro_import on the same data source before running a synchro_exec don't leave Full load interval = 0s otherwise deletion policy will do nothing.

This parameter Full load interval is ignored by a synchro_import –synchronize=1 as any object not provided within that unique import is considered as 'obsolete'

If your set a Full load interval of 7 days, and a Retention duration of 14 days, an object no more provided by the source will be deleted after 3 weeks

Attributes

Synchro Data Source Attributes

The Attributes tab

The “Attributes” tab defines:

  1. How to determine whether a new object must be created in iTop or an existing object must be associated with the replica (if the reconciliation rule is “Use Attributes” in the properties)

  2. Which attribute(s) of the associated object must be updated, and how

When a new record appears in the synchro_data_xxx table, the synchronization mechanism will try to associate this record with an existing iTop object. If one or more Attribute is marked as “Yes” in the “Reconciliation ?” column, these columns/attributes will be used to search for an iTop object that matches all these values. If one object is found, it will become associated with this record. If no object is found, a new iTop object will be created.

The definition of appropriate reconciliation criteria is the key to the proper behavior of the data synchronization. In order for the synchronization to behave properly, you must use a list of attributes that uniquely identify objects to synchronize in a reliable manner, as reconciliation criteria.

The Update column

The Update column defines if the attribute must be updated or not.

  • Checked = Yes: the attribute will be set/updated

  • Unchecked = No: the external source information for this attribute will be ignored, the existing value on the iTop object will be kept as is.

The Update Policy column

The column Update Policy defines the behavior of the updated attribute. The possible values are:

  • Locked: the attribute will be completely driven by the data synchronization and will appear as read-only (with a small lock icon next to it) in the iTop user interface,

  • Unlocked: the attribute will be updated by the synchronization (whenever the source data changes) but end-users can still modify the attribute in parallel inside iTop,

  • Initialize if empty: the synchronization will only set a value of this attribute if the value is empty in iTop. The attribute remains modifiable inside iTop.

The Reconcialiation key column

Finally, the Reconciliation key column defines how to process attributes which are references to another iTop object (i.e. Foreign keys). The supplied data can contain either the actual value of the foreign key (id) or another attribute such as the name of the object; which must uniquely identify the object.

For foreign keys, iTop does not support the reconciliation on an combination of the multiple attributes (like name AND organization). If such a need arises, then you must use an ETL or a script to perform the desired reconciliation before providing the id to iTop.

Status

The status tab

The Status tab displays information about the execution of the synchronization. If the synchronization has never been run, the tab displays the following message:

This synchro was never run. No log yet.

Otherwise the tab displays a schema similar to the one below:

Synchro Data Source Status

Synchronization history

On the left of the screen, each execution of the synchro is displayed in an historical list, showing the overall status, the start date and time of the execution.

Click on an element of this list to show the corresponding status in detail, displayed in the two columns on the right.

The column at the left displays the status for the records in the synchro_data_xxxx table:

  • New records are records which were added in the table since the previous execution of the synchro,

  • Existing records are records which were existing before and have been loaded again since the previous execution of the synchro,

  • Disappeared records are records which were not loaded since more than the full_load_interval duration.

  • Ignored records corresponds to objects no longer existing in iTop.

The column at the right displays the consequences of the synchro, in terms of modifications on the associated objects in iTop.

  • For each New record, the consequences can be either:

    •  

    The creation of the new object in iTop (Created),

    •  

    The update of an already existing object in iTop (Updated),

    •  

    The simple association with an object in iTop, which is already up-to-date (Unchanged),

    •  

    Or an Error when trying to either create or update an object in iTop (most probably because of inconsistent data).

  • For each Existing record, the consequences can be either:

    •  

    The update of the associated object (Updated),

    •  

    No change at all (Unchanged)

    •  

    Or an error when trying to update the associated object (Errors)

  • For each Disappeared record, the consequences (depending on the “Deletion rules”) can be either:

    •  

    The associated object in iTop has been deleted (Deleted),

    •  

    The associated object in iTop has been updated (Updated),

    •  

    The associated object in iTop was left unchanged (No Action),

    •  

    Or an error occurred when trying to update or delete the associated object in iTop (Errors).

Replicas

In iTop, the Replicas are used to store the status information and the relation between the raw data loaded in the synchro_data_xxx table and the associated iTop object.

The Replicas for a given data synchronization source can be displayed by clicking on the links at the top of the status tab. A Replica is displayed as shown below:

Synchro Replica

The content of the replicas is useful to understand the behavior of the data synchronization mechanism (and to get more information about the errors), but its data are maintained by the synchronization and should never be modified manually.

Using MySQL statements to populate the synchro_data_xxxx table

If you populate the table synchro_data_xxx using SQL commands (either with an ETL or a home made script), you must take care of never creating duplicate records. Remember that each record in the synchro_data_xxx table corresponds to exactly one object in iTop.

In order to avoid such duplicates, you must first query for an existing record before inserting a new record into the table. To speedup this query you can either construct a value that uniquely identifies the “source” object and store this value in the “primary_key” column of the table (which has a unique index), or use any combination of the record's column and alter the definition of the synchro_data_xxxx table to add your own indexes.

Do not use the MySQL statement INSERT … ON DUPLICATE KEY UPDATE to insert records in the synchro_data_xxx table. This is not supported by the internal iTop mechanism which relies on the “BEFORE INSERT” trigger.

Should you need to empty the synchro_data_xxxx table, do not use the MySQL statement TRUNCATE TABLE, since this statement will actually drop and reconstruct the table… but not its triggers! Instead use DELETE FROM synchro_data_xxxx.

Then trigger the execution of the data synchronization between the iTop objects and thereplicas of the temporary table by executing the page “synchro/synchro_exec.php” either in command line mode, or by invocating the web page (using wget for example).

Using synchro_import to populate the synchro_data_xxxx table

See: synchro_import.php

The synchronize parameter of synchro_import.php trigger synchro_exec.php by default

How to specify a list of related objects (link set)

Some classes have attributes called Link Sets. Example: UserLocal.profile_list

You can synchronize those lists by writing all the information in the corresponding column:

Here is an example specifying that the login will have one profile: 'Portal User'

INSERT login, profile_list INTO data_synchro_userlocal_1 VALUES ('johndoe', 'profileid->name:Portal User;reason:Customer')...;

You can also specify several links (several profiles).

More information in Import a LinkedSet

标签:
由 superadmin 在 2020/08/27, 16:21 创建
    

需要帮助?

如果您需要有关XWiki的帮助,可以联系:

深圳市艾拓先锋企业管理咨询有限公司