Get documents about "Module 3 Designing an Active Directory Site Topology
Document Sample
Module 3:
Designing an
Active Directory
Site Topology
Agenda
Sites
Replication Within Sites
Replication Between Sites
Replication Protocols
Active Directory Branch Office Deployment
What Are Sites?
The First Site Is Set Up Automatically, and Is Called
Default-First-Site-Name
Sites Can Consist of Zero, One, or More Subnets
Sites Are Used to Control Replication Traffic and
Logon Traffic
Sites Contain Server Objects and Are Associated with
IP Subnet Objects
Sites: Purpose and Function
Definition:
A set of well-connected subnets
Contain only Server and Configuration
objects
Sites are used for…
Logging on
Group Policies
Replication topology
Intra-Site
Inter-Site
Site Boundaries
Sites may span domains
Domains may span sites
OUs may span sites
Site A spans some
components of dom.com
dom.com
Site A Some components of
OU OU exist in SiteA
Some components of
OU exist in Site B
Site B
Site B spans all of sub.dom.com
sub.dom.com and
some components of
dom.com
Replication
Intra-Site Replication
Automatic topology generation
Pull-only, based on update
notification
Always RPC based
Inter-Site Replication
Semi-automatic topology generation
Scheduled (no update notification)
RPC or SMTP (Domain NC: RPC Only)
Intra-Site Replication
Information that is replicated
Domain Naming Context (NC)
Configuration Naming Context (NC)
Schema Naming Context (NC)
Replication Topologies
Domain NC
Schema/Configuration NC always
share the same topology
Intra-Site Replication
Same Site - Single Domain
A
1
SITE A
2
= One Replication
Topology
Each new DC (KCC) inserts
itself into the ring
A A Replication via RPC is
3 4
based on pull
Legend
Domain
Topology adjusts to
A
3
Server
ensure a maximum of three
Domain A Topology/Connection
hops (edges added at 7
Schema/Configuration Topology servers)
KCC runs every 15 minutes
Intra-Site Replication
A
SITE A
1 2
B
2 DCs within a site/domain
B
will maintain distinct
1 Domain NC connection
B objects
3
A
3
A
4
Schema/Configuration
replication performed
Legend normally
A Domain Domain NC’s topologies
3 Server
are separate for
Domain A Topology/Connection Domain A and
Domain B Topology/Connection
Schema/Configuration Topology
Domain B
Intra-Site Replication
Global Catalog
Servers within a site
will source from a DC
SITE
Global Catalog will
A A establish a connection
1 2
B object to request
2
Domain NC from the
other domain(s)
B Global Catalog
1 Server & Connector
B
3
A A
3 4
Inter-Site Replication
Site Links
Two or more sites
Connected by common transport
Cost associated with link
Schedule determines window
Frequency determines replication
Site Link Bridges
Two or more site links
Transitiveness between site links
Site Links
Transport
IP (RPC) or SMTP
Cost
Smaller number is cheaper
Based on network characteristics
Schedule
Configurable
Schedule defines windows of
replication
Frequency defines how often
replication will happen
Site Links
Describe physical
NYC network
AB 20
1
FRAME
Used for message
RED BOS
AB
128 128
AB route paths
Defined by:
256
10
10
Two or more sites
SEA ATL
A
A 512
AB Cost
5
Transport
512
5
Legend
LAX COST Schedule
B Site LInk Domain
B Frequency
A Domain Servers
B
Site Link Bridges
NYC
AB 2
1
0
FRAME BOS
Provide
RED 128128
AB AB transitiveness
between site links
256
10
10
SEA ATL
Similar to network
A
A 512
AB routers
5
All sites bridged by
512
5
LAX
default
B
B Defined by two or
more site links
Replication Within Sites
Domain
Controller A
Site
IP Subnet
Replication
IP Subnet Domain
Controller B
Replication Within Sites:
Occurs Between Domain Controllers in the
Same Site
Assumes Fast and Highly Reliable Network
Links
Does Not Compress Replication Traffic
Uses a Change Notification Mechanism
Replication Between Sites
Replication Between ISTG
Bridgehead
Sites: Server
Replication
Occurs on a
Manually Defined IP Subnet
Schedule Site IP Subnet
Is Designed to
Optimize Bandwidth Replication
One or More
Replicas in Replication
Each Site Act As Bridgehead
Bridgeheads IP Subnet Server, ISTG
Site IP Subnet
Replication Protocols
RPC or SMTP
Domain Controller A Domain Controller B
Replication Protocols
RPC for Replication Within and Between
Sites
SMTP for Replication Between Sites
ISM and KCC/ISTG
Inter-Site Messaging Service (ISM)
Creates cost matrix for Inter-Site replication
Sends and receives SMTP messages if SMTP
replication is used
Runs only when
ISM service starts up
Changes happen in site configuration (new
sites, site-links, site-link-bridges)
Information is used by
Netlogon for auto-site coverage
Load-Balancing tool
Universal Group Caching
DFS
KCC/ISTG
Computes least cost spanning tree Inter-Site
replication topology
Inter-Site component of KCC
Runs every 15 minutes by default
Bridgehead Server Selection
Windows 2000
On a per site basis, for each domain, one DC
per NC used as Bridgehead
Windows Server 2003
On a per site basis, for each domain, all DCs
per NC used as Bridgehead
KCC picks DC randomly when connection
object is created
For both incoming and outgoing connection
objects
Bridgehead Server Selection
A
B A1 A2 A3 B1 B2 B3 B4
siteLink
siteLink siteLink
A11 B11 A12 B12 A13 B13
Bridgehead Server Selection
Windows 2000
A
B A1 A2 A3 B1 B2 B3 B4
A11 B11 A12 B12 A13 B13
Bridgehead Server Selection
Preferred Bridgehead Server List
Some servers should not be used as
Bridgeheads
PDC FSMO
Weak hardware
Solution: Preferred Bridgehead Server List
Allows administrator to restrict what DCs can
be used as Bridgehead Servers
If Preferred Bridgehead Server List is defined
for a site, KCC/ISTG will only use members of
the list as Bridgeheads
Warning:
If Preferred Bridgehead Server List is defined,
make sure that there are at least DCs per NC
in the list
If there is no DC for a specific NC in the list,
replication will not occur out of site for this
NC
Bridgehead Server Selection
Preferred Bridgehead Server List
A
B A1 A2 A3 B1 B2 B3 B4
A11 B11 A12 B12 A13 B13
Bridgehead Server Selection
Bad Bad Preferred Bridgehead Server List
A
B A1 A2 A3 B1 B2 B3 B4
Replication
to B NC
broken
A11 B11 A12 B12 A13 B13
Bridgehead Server Selection
Recommendations for Branch Office Deployments
Always use Preferred Bridgehead Server
List in hub sites
Make sure that there are enough DCs in the
list
Make sure that there are enough DCs that
are not included in the list
Do not add PDC Operations Master
Do not add DCs used for user logons
Do not add DCs used by Exchange servers
Make sure that all NCs are covered in the
Preferred Bridgehead Server List
If there are GCs in the branches, make all
Bridgehead Servers GCs
Best Practices
Place at Least One Domain Controller in Every Site
Place At Least One DNS Server in Each Site
Schedule Site Links for Times When Network Traffic Is Slow
Agenda
Sites
Replication Within Sites
Replication Between Sites
Replication Protocols
Active Directory Branch Office Deployment
Characteristics Of A Branch
Office Deployment
Large number of locations
Small number of users per location
Hub and spoke network topology
Slow network connections and dial
on demand links
WAN availability
Bandwidth available for Active Directory
Other services relying on the WAN
Large number of domain controllers
in remote locations
Branch Office Scenario
TCP/IP & DNS Settings
AD Branch Office Scenario FSMO Role Placement
ROOT1- GC ROOT2 - DC
(SM, DNM FSMO Roles) (IM, RID, PDC FSMO Roles)
corp.hay-buv.com corp.hay-buv.com ROOT3 - DC
10.10.1.1 10.10.1.2 corp.hay-buv.com
DNS P: 10.10.1.2 DNS P: 10.10.1.1 10.10.1.3
DNS A: 10.10.1.3 DNS A: 10.10.1.3 DNS P: 10.10.1.1
DNS A: 10.10.1.2
HUBDC1 - DC
(IM, RID, PDC FSMO Roles)
branches.corp.hay-buv-com
10.10.20.99
DNS P: 10.10.20.99
BH3 - GC
DNS A: 10.10.20.1 BH1 - GC
BH2 - GC branches.corp.hay-buv.com
branches.corp.hay-buv-com
branches.corp.hay-buv.com 10.10.20.3
10.10.20.1
10.10.20.2 DNS P: 10.10.20.3 Site-HUB
DNS P: 10.10.20.1
DNS P: 10.10.20.2 DNS A: 10.10.20.1
DNS A: 10.10.20.2
DNS A: 10.10.20.1
Staging - GC
branches.corp.hay-buv.com
10.10.30.1
DNS P: 10.10.30.1
DNS A: 10.10.20.1
Site-Stage
Branch Office Branch Office Branch Office Branch Office Branch Office
Site-Branch1 Site-Branch2 Site-Branch3 Site-Branch4 Site-Branch5
BODC1 BODC2 BODC3 BODC4 BODC5
DC DC DC DC DC
10.10.21.1 10.10.22.1 10.10.23.1 10.10.24.1 10.10.25.1
DNS P: 10.10.21.1 DNS P: 10.10.22.1 DNS P: 10.10.23.1 DNS P: 10.10.24.1 DNS P: 10.10.25.1
DNS A: 10.10.20.1 DNS A: 10.10.20.2 DNS A: 10.10.20.3 DNS A: 10.10.20.1 DNS A: 10.10.20.2
Design Considerations For Branch
Offices
User management and Group Policies
Structural Planning
Forest planning
Domain planning
DNS considerations
Replication planning
DNS configuration for branch offices
Replication planning
Centralized User Management
Advantages:
Good security control and policy enforcement
Easy automation of common management tasks from a single
source point
Problems can be fixed quickly
Changes flow from hub to branch
Disadvantages
Success varies directly with the availability and speed of the
local area network (LAN) or WAN
Propagation changes are time-consuming, depending on the
replication infrastructure and the replication schedules
Time to react and to fix issues might be longer
IT organization tends to be further away from “customer”
Recommendation
Use centralized model
Group Policy Management
Management of Group Policies focuses on PDC
Group policies use both Active Directory and
sysvol replication (NTFRS replication)
Sysvol replicates on a per file level
Changes are performed on PDC
Always use centralized Group Policy model for
Branch Office deployments
Watch applications that change Group Polices
(account security settings)
Restrict administration of policies to group that
understands impact of changes
Avoid last writer win overwrite issues
SYSVOL Replication
Follows AD replication topology
Uses connection objects
Different conflict resolution algorithm
Replicates on a per file level
Last writer wins
Avoid applications that create excessive
sysvol replication
Do not create file system policy against
replicated content
Check anti-virus software
Diskeeper
Forest Planning
Deploy single forest for Branch Offices
Reasons for having multiple forests
Political/organizational reasons
Unlikely in branch office scenarios
Too many locations where domain controllers
must be deployed
Complexity of deployment
Too many objects in the directory
Should be partitioned on domain level
GCs too big?
Evaluate not deploying GCs to branch
offices
“Whistler”: No-GC-logon feature
Domain Partitioning
Recommendation for Branch Office Deployment
Use single domain
Typically only one security area
Central administration (users and policies)
Replication traffic higher, but more flexible
model (roaming users, no GC dependencies)
Database size no big concern
If high number of users work in central location
Create different domains for headquarters and
branches
If number of users very high (> 50,000)
Create geographical partitions
Design Considerations For Domain
Controller Placement
Required services
File and Print, e-mail, database, mainframe
Most of them require Windows® logon
Logon requires DC and GC availability
Logon locally or over the WAN
WAN logon requires acceptable speed
and line availability
Cached credentials only work for local
workstation logon
Design Considerations For Domain
Controller Placement
Replication versus client logon traffic
Replication traffic more static and predictable
Affected by domain design and GC location
Applications using the GC can demand local
GC
Logon traffic affected by number of users in the
branch and services
Less predictable
Security
Management
Alternative solutions
Terminal Servers
Local accounts
Design Considerations For Global
Catalog Placement
No factor in single domain deployment
Multiple Domain deployments
GC needed for logon in native mode
Disable GC requirement
“Whistler” has Universal Group caching feature
Services might require GC
Exchange 2000
Recommendation:
If WAN unreliable or more than 50 users in branch,
deploy GC to branch
Always put GC next to services that require GC
I.e., if there are Exchange 2000 servers in the branch,
deploy GC to branch
DNS Planning Considerations
DNS – AD root domain
Distributing forest wide locator records
Island problem
Domain controller SRV record configuration
Auto Site Coverage
NS records
DNS Configuration
Of Root Domain
If DNS already exists
Delegate AD root domain to Windows 2000
DNS server (i.e., corp.microsoft.com)
Use Active Directory integrated DNS zones
If not
Use Windows 2000 DNS server on domain
controllers
Use Active Directory integrated DNS zones
Create internal root, or Configure forwarders
Distributing Forest Wide Records
CNAME records for replication and GC records are forest
wide records
Stored in _msdcs domain in the AD root domain
I.e., two domains, corp.microsoft.com and
sales.corp.microsoft.com
A records for DCs in corp.microsoft.com are stored
in the corp.microsoft.com DNS domain
A records for DCs in sales.corp.microsoft.com are
stored in sales.corp.microsoft.com
CNAMEs for replication for DC in corp.microsoft.com
are stored in the _msdcs.corp.microsoft.com DNS
domain
CNAMEs for replication for DC in
sales.corp.microsoft.com are stored in the
_msdcs.corp.microsoft.com DNS domain
By default, this domain exists only on root domain
controllers
Create separate zone for _msdcs.<ForestRootDomain>
and transfer zone to all DCs in child domains
The Island Problem
A domain controller that is also a DNS server can isolate
itself from replication
Can only happen if
DC points to itself as preferred or alternate DNS
server
DC has writeable copy of
_msdcs.<forestRootDomain> DNS domain
Recommendation
Domain controllers that are DNS servers AND are
domain controllers in the forest root domain should
point to another DC as preferred and alternate
DNS server
All other domain controllers (especially child domain
controllers) can point to themselves as preferred or
alternate DNS server
Managing Service Records
SRV records are published by netlogon in DNS
On site level and domain level
Clients search for services in the client site first,
and fall back to domain level
Branch Office deployments require specific configuration
Large number of domain controllers creates scalability problem
for domain level registration
If more than 850 branch office DCs want to register SRV records
on domain level, registration will fail
Registration on domain level is in most cases meaningless
DC cannot be contacted over WAN / DOD link anyways
If local look-up in branch fails, client should always fallback to
hub only
Configure netlogon service to register SRV records for Branch Office
DCs on site level only
Follow Q267855
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netl
ogon\Parameters
Registry value: DnsAvoidRegisterRecords
Data type: REG_MULTI_SZ
Mnemonic Type DNS record
Dc SRV _ldap._tcp.dc._msdcs.<DnsDomainName>
_ldap._tcp.<SiteName>._sites.dc._msdcs.<DnsDomainNam
DcAtSite SRV e>
_ldap._tcp.<DomainGuid>.domains._msdcs.<DnsForestNa
DcByGuid SRV me>
Pdc SRV _ldap._tcp.pdc._msdcs.<DnsDomainName>
Gc SRV _ldap._tcp.gc._msdcs.<DnsForestName>
GcAtSite SRV _ldap._tcp.<SiteName>._sites.gc._msdcs.<DnsForestName
GenericGc SRV _gc._tcp.<DnsForestName>
GenericGcAtSite SRV _gc._tcp.<SiteName>._sites.<DnsForestName>
GcIpAddress A _gc._msdcs.<DnsForestName>
CNA
DsaCname ME <DsaGuid>._msdcs.<DnsForestName>
Kdc SRV _kerberos._tcp.dc._msdcs.<DnsDomainName>
_kerberos._tcp.dc._msdcs.<SiteName>._sites.<DnsDomain
KdcAtSite SRV Name>
Ldap SRV _ldap._tcp.<DnsDomainName>
LdapAtSite SRV _ldap._tcp.<SiteName>._sites.<DnsDomainName>
LdapIpAddress A <DnsDomainName>
Rfc1510Kdc SRV _kerberos._tcp.<DnsDomainName>
Rfc1510KdcAtSite SRV _kerberos._tcp.<SiteName>._sites.<DnsDomainName>
Rfc1510UdpKdc SRV _kerberos._udp.<DnsDomainName>
AutoSite Coverage
AutoSite coverage allows DCs to
advertise for sites without DCs, if they
are in the closest site to the DC
Not practical for Branch Office
deployments
Root DCs would advertise for all sites
If client cannot connect to a local DC,
it will fall back to hub site anyways
(configuration of SRV records)
Planning For Replication
Concepts
Connection objects
KCC
Site-links
Site-link bridges
Sysvol replication
Planning steps
Planning for Bridgehead Servers
Determine number of Sites
Decide whether to use the KCC or create replication
topology manually
Define site structure of hub site
Define replication schedule
Create Site-Links
Create connection objects (if KCC disabled)
Planning For Bridgehead
Servers
How many bridgehead servers do I need?
How to configure bridgehead servers
Things you need to know:
Centralized or decentralized change model
Data update requirements formulated by customer
How many times a day do we need to replicate?
How many changes happen in a branch per day
Total number of domain controllers
Time needed to establish dial-on-demand
network connectivity
Inbound Versus Outbound
Replication
Different threading model
Outbound replication is multi-threaded
Bridgehead server can have multiple
replication partners
Bottleneck is most likely CPU (monitor!)
Inbound replication is single-threaded
Replication of changes from branches
to hub is serialized
Bottleneck is most likely the network
Replication Traffic
Documented in Notes from the Field,
Building Enterprise Active Directories
Replication overhead for branch office deployments
Overhead if there are two domain controllers:
21 KB
13 KB to setup the replication sequence
5 KB to initiate replication of the domain naming
context, including the changed password
1.5 KB for each schema and configuration
naming context (where no changes occurred)
Each DC will add 24 Bytes
Overhead for 1,002 DCs: 162 KB
Example – Outbound Replication
Partners
Requirements:
Replication twice a day (= K)
WAN 8 hours available (= H)
High performance hardware (= 30 concurrent
connections) (= O)
Outbound replication will always finish within 1
hour (= T)
Applying the formula:
OC = (H * O) / (K * T) = (8 * 30) / (2 * 1) = 120
Each bridgehead server can support 120 branch
office DCs (outbound)
If number is too high/low, change parameters:
I.e., WAN available for 12 hours: 180 branches
I.e., replicating only once a day: 240 branches
Example – Inbound Replication
Partners
Let’s assume slow WAN with DOD lines
Factors
Replication traffic (time to submit changes
like password changes)
Time to setup DOD connections
4 minutes per branch is conservative
IC = R / N = 480 / 4 = 120 Branches
Example – Inbound Replication
Partners
Number of branches supported by one bridgehead
servers is lower value of results
Outbound: 120 branches
Inbound: 120 branches
Result: One bridgehead can support 120 branches
If you have 1,200 branches, you need 10 bridgehead
servers
Plan for disasters and special cases!
Leave headroom for hub outbound replication
Have spare machine available
Create multiple connections from branch to hub DCs
Bridgehead Server Overload
Symptoms
Bridgehead cannot accomplish replication requests
as fast as they come in
Replication queues are growing
Some DCs NEVER replicate from the bridgehead
Once a server has successfully replicated from
the bridgehead, its requests are higher prioritized
than a request from a server that has never
successfully replicated
Monitoring
Repadmin /showreps shows NEVER on last
successful replication
Repadmin /queue <DCName>
Bridgehead Server Overload
Can be caused by
Unbalanced site-links (if ISTG is enabled)
Unbalanced connection objects
Replication schedule too aggressive
Panic trouble-shooting
Like changing replication interval on all site-links
to a drastic shorter interval to accommodate
applications
Solution
If ISTG is enabled
Turn off ISTG (prevent new connections from
being generated)
Delete all inbound connection objects
Correct site-link balance and schedule
Enable ISTG again
Bridgehead Server Hardware
Processor
Dual/quad Pentium III or Xeon recommended for
bridgehead servers and servers supporting large
numbers of users
Memory
Minimum of 512 MB
Disks
Configure the operating system and logs on separate
drives that are mirrored. Configure directory database
on Redundant Array of Independent Disks (RAID) 5
or RAID 0+1
Use larger number of smaller drives for
maximum performance
Drive capacity will depend on your specific
requirements
Determine Number Of Sites
Rule for creating sites:
For each physical location that has WAN
connection (less than 10 MBit) to hub
If there is a DC in the location
Create a new site
If not, if there is a service that uses the site model
(DFS shares)
Create a new site
If not, create subnet for the location and add subnet
to hub site (or next closest site)
Use Of KCC For Inter-Site Replication
Topology Generation
Always disable transitiveness
Windows 2000:
Less than 500 sites: Use KCC
But test your hardware first
Follow guidelines in KB article Q244368
More than 500 sites: Create connection
objects manually
Branch Office deployment guide
recommends manual topology for more
than 100 sites
Windows Server 2003: Use KCC
Define Site Structure Of Hub Site
If KCC is disabled, create single site
If KCC is enabled, create one site per
Bridgehead Server
KCC has no concept of Inter-Site load
balancing between servers in one site
Create artificial sites in hub site to spread load
between Bridgehead Servers
Create Site-Links with staggered schedules
between branches and hub sites
Load Balancing With Sites
Site-link: Schedule 2am – 4am, cost 100
Site-link: Schedule 4am – 6am, cost 100
Site-link: Schedule: Always (Notification enabled), cost 1
Hub Site-Link
SiteA SiteB SiteC
BranchC2
BranchA1
BranchA2 BranchB1 BranchB2 BranchC1
Load Balancing Manually
(With Redundancy)
Replicate on
alternating
schedule Hub Site
DC1 DC2 DC3
Branch1 Branch6
Branch2 Branch5
Branch3 Branch4
Building The Hub Site
Building the root domain
Availability of root domain
Only needed for special configuration tasks
Adding new domains
Schema changes
Kerberos trusts and dynamic registration of forest
wide resource records might depend on root
domain
Operations Masters
Typically not critical for root domain
Server Sizing
Empty root domain does not require high-end
hardware
Kerberos referrals and dynamic DNS updates
Disaster Recovery
Root is critical for forest
Make sure that you perform regular backups
Building The Hub Site
Building the Branch Office Domain
Operations Master
Off-load PDC operations master
Move Infra-structure master off GC
RID master is the most critical operations master
- monitor this machine very closely
Bridgehead Servers
If Branch Office DC are GCs, then Bridgehead
Servers should be GCs
If DNS runs on Branch Office DCs, don’t run DNS
on Bridgehead Servers
Disaster recovery
State on Bridgehead server not very
interesting - not an ideal candidate for backup
Leave headroom on Bridgehead, or have
spare machine in place
Staging Site
Most companies use outsource partners to
build servers and domain controllers
Machines are built at the factories
Server usually build from image and promoted
later
Where to promote domain controllers
Staging site: Less network traffic, better
control of process, opportunity to run QA
scripts while machine is accessible
In branch: Configuration less complex
(domain controller finds its site)
Building The Staging Site
Staging Site needs to be permanently connected
to production environment
New DCs must be advertised
New DCs need RID pool
Fully control replication topology in the staging site
Only case where KCC should be disabled for Intra-
Site replication topology generation
Reason is that once machines are moved out,
domain controllers that have not learned that will
try to replicate or re-route (DOD lines)
Capacity planning for domain controller used as
source
Usually not a high-end machine
Depends on how many DCs are installed in parallel
Software installation
Add Service Packs and QFEs to image
Include Resource Kit, Support Tools and scripts for
management and monitoring
Document what is loaded on DC before
machine is shipped
Domain Controller Build Process
Use dcpromo answer file to promote the domain
controllers
Do not turn off DCs before shipping them
Best practice is to build DCs when they are needed,
not months before
If they are off-line for too long, they get out-of-sync
with production
Tombstone lifetime
Domain controller passwords
Install monitoring tools and make sure that monitoring
processes are in place
Configure domain controller for new site
Clean-up old connection objects before shipping
the machine
React if you find issues with domain controllers during
the deployment
Don’t keep processes in place if they are broken
General Considerations For
Branch Office Deployments
Ensure that Your Hub is a Robust Data Center
Do Not Deploy All Branch Office Domain
Controllers Simultaneously
Monitor load on Bridgehead servers as more and
more branches come on-line
Verify DNS registrations and replication
Balance Replication Load Between Bridgehead
Servers
Keep Track of Hardware and Software Inventory
and Versions
Include Operations in Your Planning Process
Monitoring plans and procedures
Disaster recovery and troubleshooting strategy
Personnel assignment and training
Personnel Assignment and Training
