Raval Fichadia

John Wiley & Sons, Inc. 2007

Operating Systems
Security
Chapter Seven
Prepared by: Raval, Fichadia

The Big Picture

Elements of an operating
system environment.
Some risks that impact the
operating system.

Operating system primer

Operating system: the system software on a machine upon
everything else depends.

Most critical software relied upon by almost every other

application on a machine.

Key responsibilities of an OS include:

CPU/process management

Memory management

Hardware management

Disk and file system management

Examples of popular operating systems include

Windows, Unix, Linux, zOS, MacOS.
4

Operating system primer

Operating system: the system software on a machine
upon everything else depends.

Orange book describes the criteria for security

ratings on an operating system.
Division D (Minimum Security)
Class D Systems that arent rated higher
Division C (Discretionary Protection)
Class C1 Discretionary security protection (C1 is no longer used to rate OSs)
Class C2 Controlled access protection (rated versions include versions of OpenVMS,
Windows, AS/400, RS/6000)
Division B (Mandatory Protection)
Class B1 Labeled security protection (rated versions include DEC, HP-UX, IRIX)
Class B2 Structured protection (rated versions include XENIX 4.0)
Class B3 Security domains (rated versions include XTS-200)
Division A (Verified Protection)
5
Class A1 Verified design (rated versions include Boeing MLS)

Management concerns
Concerns about operating system security typically include
the following:

Minimizing the companys risk exposure in the most

cost-effective manner.

Defining the optimal security posture across a

heterogeneous environment.

Keeping up with existing and upcoming security threats

against operating systems and implementing mitigating
controls.

Having an effective backup, recovery, business

resumption and a disaster recovery plan.
6

Risks and controls

Authentication: Process of validating user identity

Most operating systems require a user ID and a

password combination for authentication.

Some allow for third-party based authentication.

User IDs are used by regular end users, administrators,

and applications to claim their identity.

Knowledge of the password proves the users identity.

Operating systems typically never store passwords,

rather they store the password hash - a unique fixed
length output based on the password and/or a salt.

Risks and controls

Authentication: Process of validating user identity

Wrong password generate wrong hashes and user is

denied access. Right password generates right hash
and allows user access.

Risks and controls

Authentication risks:

Poor password selected by end users.

Dictionary and/or brute-force attacks against password

hashes.

Controls:

Password management routines such as requiring

periodic password changes, quality passwords, longer
passwords, etc.

Enabling account lockouts after a given number of

wrong password attempts.

Using two-factor authentication techniques: something

you have and something you know.
9

Risks and controls

Authorization: Process of determining user privileges

Most operating systems determine user IDs privileges

after users authentication.

Protected resources have an Access Control List (ACL)

that defines who has what level of access.

User IDs are assigned privileges to protected resources

either directly, or by them being part of a group that has
access.

Some operating system like Unix/Linux allow users to

exceed their privileges via features like setuid/setgid.
10

Risks and controls

Authorization risks:

Poorly defined ACLs.

Attacks that let users again unauthorized privileges

(setuid/setgid exploits, exploiting services/applications
with higher privileges).

Controls:

Periodic review of ACLs and privileges granted to users

and groups.

Protecting setuid/setgid related files from exploited by

implementing stronger file permissions.

Disabling unnecessary privileged services / uninstalling

unnecessary privileged applications.
11

Risks and controls

Trust relationships: Relying on other OS for validating a
user identity.

Many operating systems allow access to their resources

without verifying a users credentials if they trust other
OS of doing so.

Trust simplifies and adds convenience for user and

application by not having to reauthenticate.

Without trust relationships credentials may have to be

stored in multiple places.

Trust can be unidirectional or bidirectional.

12

Risks and controls

Trust risks:

Spoofing attacks wherein users forge their identities.

Transitive trust that let users go from AB and from

BC, however AC wasnt intended.

Controls:

Periodic review of trust relationships and their business

need. Disable the ones that arent needed.

Implement patches and controls to prevent spoofing

attacks.

13

Risks and controls

Job scheduling: Allows programs and scripts to run without
user interaction at scheduled times.

Most operating systems allow definition of jobs

(programs, scripts, batch files) to be run periodically at
scheduled hours with different level of privileges.

Schedulers can offload backups during off-peak hours.

Jobs can be scheduled at different hours, or upon

different triggers such as system startup, shutdown,
results of previous job etc.

14

Risks and controls

Job scheduling risks:

Piggybacking attacks wherein malicious code is

inserted into unprotected scripts (or scripts referenced
by the job) that are scheduled as privileged jobs.

Program substitution wherein a malicious script is

executed instead of the original due to lax permissions
on directories or because of lack of full path names.

Controls:

Periodic review of all jobs and their privileges. Disable

the ones that arent needed and/or minimize privileges.

Implement strong file and directory permissions.

15

Risks and controls

File systems: OS feature to manage files stored on the hard
drive(s).

File systems allow for tracking of the location of files

and their directories, their classification, their attributes,
their ACLs, and their security.

File systems can be for local files (files stored on locally

on the computer) or for remote files (files shared with
remote computers).

OSs can support multiple types of file systems such as

NTFS, FAT, EFS (local), NFS, CIFS (remote) etc.

Local and remote file systems are protected via ACLs

that define who has what type of access to the file.

16

Risks and controls

File systems: OS feature to manage files stored on the hard
drive(s).

Most file systems classify files in an inverted tree

structure.

17

Risks and controls

File system risks:

Default ACLs or user-defined ACLs on local and shared

files could be poor.

Critical files containing sensitive/confidential data could

be poor secured.

Controls:

Periodic review ACLs for sensitive files and directories.

Use encryption on sensitive files and directories.

Baseline the file system to detect unauthorized

changes.

Periodic review ACLs for shared files.

18

Risks and controls

Software updates: Updating OS with vendor provided fixes.

Operating systems have defects that are identified

either by end users or by vendors.

Vendors release fixes patches to fix/mitigate the

defect.

Hotfixes are patches that have limited regression testing

done and typically address a small number of issues.

Service packs are collection of patches that are

regression tested and address several issues.

19

Risks and controls

Software update risks:

Inadequate testing of patches by vendor or end-user

could break existing functionality.

Zero-day exploits could hit OS before patches are

applied.

Controls:

Periodic review threats and their fixes.

Maintain inventory of OS and their patch level.

Have a test bed ready to test the impact of patches on

existing functionality.

Use automated tools to deploy patches at the earliest.

20

Assurance considerations
An audit to assess operating system security should include
the following:

Evaluate the need for each service on the OS.

Unnecessary services should be turned off.

Assess the patch deployment process. All necessary

patches should be implemented.

Review authentication methods, user accounts and their

privileges. Ensure quality user ID and password
management techniques are used.

Determine the legitimacy of trust relationships.

Ensure core operating system files are baselined and

protected.

21

Assurance considerations

Evaluate the need and security of all file shares.

Determine if additional controls like anti-virus, personal

firewalls, and intrusion prevention systems are in place.

Review all scheduled jobs and their controls.

Ensure that functional plans for backup and recovery,

business resumption, disaster recovery are in place.

22

Recap

23