runbook init
This commit is contained in:
@@ -39,3 +39,32 @@ MYSQL_USER=nextcloud
|
|||||||
|
|
||||||
NEXTCLOUD_ADMIN_USER=admin
|
NEXTCLOUD_ADMIN_USER=admin
|
||||||
NEXTCLOUD_ADMIN_PASSWORD=DesI$8w9ad_4
|
NEXTCLOUD_ADMIN_PASSWORD=DesI$8w9ad_4
|
||||||
|
|
||||||
|
## User Accounts
|
||||||
|
admin
|
||||||
|
Death0fP@n
|
||||||
|
|
||||||
|
Ken Schaefer
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Gregg Williams
|
||||||
|
gw@gaussianwellworks.com
|
||||||
|
2qj2ky6q
|
||||||
|
bra=o5ABr!
|
||||||
|
|
||||||
|
Ruud Weijermars
|
||||||
|
rw@gaussianwellworks.com
|
||||||
|
q9v81x0r
|
||||||
|
C9E#ut=PEf
|
||||||
|
|
||||||
|
Khaled Raeeini
|
||||||
|
kr@guassianwellworks.com
|
||||||
|
n59fejcr
|
||||||
|
zeQu3eF7&k
|
||||||
|
|
||||||
|
Lisa Jack
|
||||||
|
lj@gaussianwellworks.com
|
||||||
|
8v3wtbfn
|
||||||
|
t9proM8fI+
|
||||||
|
|
||||||
|
|||||||
584
Runbooks/System Admin Runbook.md
Normal file
584
Runbooks/System Admin Runbook.md
Normal file
@@ -0,0 +1,584 @@
|
|||||||
|
# Fractional Insight CIO
|
||||||
|
## Operational Runbook — Daily & Weekly Administration
|
||||||
|
### DigitalOcean / Docker / Nextcloud / Fastmail Pilot Environment
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 1. Purpose
|
||||||
|
|
||||||
|
This operational runbook defines the recurring administrative tasks required to safely operate and maintain the client pilot environment hosted on DigitalOcean infrastructure using:
|
||||||
|
|
||||||
|
- Nextcloud
|
||||||
|
- Fastmail
|
||||||
|
- Docker containers
|
||||||
|
- Linux server administration
|
||||||
|
- Reverse proxy / SSL management
|
||||||
|
- Backup and recovery validation
|
||||||
|
- Security and compliance oversight
|
||||||
|
|
||||||
|
The goal of this runbook is to:
|
||||||
|
|
||||||
|
- Reduce operational risk
|
||||||
|
- Reduce exposure to liability
|
||||||
|
- Detect security incidents early
|
||||||
|
- Ensure recoverability of client data
|
||||||
|
- Maintain stable uptime and user access
|
||||||
|
- Establish evidence of reasonable administrative diligence
|
||||||
|
|
||||||
|
This document assumes:
|
||||||
|
- Remote users
|
||||||
|
- No internal IT staff
|
||||||
|
- Small pilot deployment
|
||||||
|
- Shared responsibility model between consultant and client
|
||||||
|
- MFA enforcement in both Fastmail and Nextcloud
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 2. Operational Philosophy
|
||||||
|
|
||||||
|
The environment should be treated as:
|
||||||
|
|
||||||
|
- A business collaboration platform
|
||||||
|
- A controlled data environment
|
||||||
|
- A security-sensitive system
|
||||||
|
- A system requiring documented administrative oversight
|
||||||
|
|
||||||
|
Because the platform contains:
|
||||||
|
- Client communications
|
||||||
|
- Potential confidential documents
|
||||||
|
- Shared file repositories
|
||||||
|
- User credentials
|
||||||
|
- Internet-exposed services
|
||||||
|
|
||||||
|
…administration must prioritize:
|
||||||
|
|
||||||
|
1. Security first
|
||||||
|
2. Recoverability second
|
||||||
|
3. Stability third
|
||||||
|
4. Convenience last
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 3. Daily Operational Tasks
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3.1 Morning Health Check
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Daily (business days)
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
10–15 minutes
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
Confirm that all core systems are operational before users begin work.
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
#### Infrastructure
|
||||||
|
- Verify droplet online in DigitalOcean
|
||||||
|
- Verify CPU/RAM/disk usage within normal thresholds
|
||||||
|
- Verify disk utilization below 80%
|
||||||
|
- Verify Docker daemon operational
|
||||||
|
|
||||||
|
#### Services
|
||||||
|
- Verify Nextcloud web login functional
|
||||||
|
- Verify Fastmail operational status
|
||||||
|
- Verify SSL certificates valid
|
||||||
|
- Verify reverse proxy routing functional
|
||||||
|
|
||||||
|
#### Containers
|
||||||
|
Check:
|
||||||
|
- Nextcloud container
|
||||||
|
- Database container
|
||||||
|
- Redis container
|
||||||
|
- Reverse proxy container
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```bash
|
||||||
|
docker ps
|
||||||
|
```
|
||||||
|
|
||||||
|
#### External Access Test
|
||||||
|
Validate:
|
||||||
|
- HTTPS access
|
||||||
|
- File upload/download
|
||||||
|
- Login functionality
|
||||||
|
|
||||||
|
#### Email
|
||||||
|
Send/receive test email through Fastmail test account.
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Daily operational log entry
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3.2 Security Event Review
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Daily
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
10 minutes
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
Identify suspicious activity before escalation.
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
#### Review:
|
||||||
|
- Failed login attempts
|
||||||
|
- MFA failures
|
||||||
|
- New device logins
|
||||||
|
- Suspicious IP addresses
|
||||||
|
- Excessive upload activity
|
||||||
|
- Unexpected admin actions
|
||||||
|
|
||||||
|
#### Check:
|
||||||
|
- Nextcloud security warnings
|
||||||
|
- Linux auth logs
|
||||||
|
- Docker errors
|
||||||
|
- Reverse proxy logs
|
||||||
|
|
||||||
|
Example:
|
||||||
|
```bash
|
||||||
|
sudo journalctl -p 3 -xb
|
||||||
|
```
|
||||||
|
|
||||||
|
### Escalation Triggers
|
||||||
|
Immediate escalation if:
|
||||||
|
- Multiple failed admin logins
|
||||||
|
- MFA bypass suspicion
|
||||||
|
- Unknown admin account
|
||||||
|
- Malware/ransomware indicators
|
||||||
|
- Unexpected outbound traffic
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Security review noted in operational log
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3.3 Backup Verification
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Daily
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
5–10 minutes
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
Verify backups completed successfully.
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
#### Verify:
|
||||||
|
- Scheduled backup job completed
|
||||||
|
- Backup storage reachable
|
||||||
|
- Backup size reasonable
|
||||||
|
- No corruption warnings
|
||||||
|
- Snapshot success in DigitalOcean
|
||||||
|
|
||||||
|
#### Validate:
|
||||||
|
- Latest backup timestamp
|
||||||
|
- Database dump presence
|
||||||
|
- File archive generation
|
||||||
|
|
||||||
|
### Important
|
||||||
|
A backup that has not been validated should be treated as nonexistent.
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Backup verification entry in operational log
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3.4 User Administration Review
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Daily
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
5–10 minutes
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
Ensure user/account integrity.
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
#### Review:
|
||||||
|
- New user requests
|
||||||
|
- Disabled users
|
||||||
|
- Terminated personnel
|
||||||
|
- Permission changes
|
||||||
|
- Shared folder permissions
|
||||||
|
- Public links
|
||||||
|
|
||||||
|
#### Verify:
|
||||||
|
- No orphaned admin accounts
|
||||||
|
- MFA enabled for all admins
|
||||||
|
- Least-privilege principles maintained
|
||||||
|
|
||||||
|
### High-Risk Areas
|
||||||
|
- Shared folders with external access
|
||||||
|
- Public upload links
|
||||||
|
- Administrative delegation
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Access review note
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3.5 Incident Queue Review
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Daily
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
5–15 minutes
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
Identify unresolved operational or security issues.
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
Review:
|
||||||
|
- User tickets
|
||||||
|
- Error reports
|
||||||
|
- Sync failures
|
||||||
|
- Email delivery issues
|
||||||
|
- Storage complaints
|
||||||
|
- Permission problems
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Updated incident tracking
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 4. Weekly Operational Tasks
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4.1 Operating System Updates
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Weekly
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
30–60 minutes
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
Maintain security posture and system stability.
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
#### Linux Updates
|
||||||
|
```bash
|
||||||
|
sudo apt update
|
||||||
|
sudo apt upgrade
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Docker
|
||||||
|
- Update container images
|
||||||
|
- Rebuild containers if necessary
|
||||||
|
- Remove unused images
|
||||||
|
|
||||||
|
#### Validate:
|
||||||
|
- Nextcloud functionality after updates
|
||||||
|
- Database connectivity
|
||||||
|
- Reverse proxy operation
|
||||||
|
|
||||||
|
### Important
|
||||||
|
Do NOT apply major-version upgrades during business hours.
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Patch log
|
||||||
|
- Change log entry
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4.2 Nextcloud Maintenance Review
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Weekly
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
20–30 minutes
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
#### Review:
|
||||||
|
- Security warnings
|
||||||
|
- Integrity check results
|
||||||
|
- App updates
|
||||||
|
- Background jobs
|
||||||
|
- Storage consumption
|
||||||
|
|
||||||
|
#### Validate:
|
||||||
|
- Cron jobs functioning
|
||||||
|
- File scanning healthy
|
||||||
|
- No database corruption warnings
|
||||||
|
|
||||||
|
#### Execute
|
||||||
|
```bash
|
||||||
|
docker exec -it nextcloud-app php occ status
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Weekly maintenance report
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4.3 Backup Restore Test
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Weekly
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
30–60 minutes
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
Prove recoverability.
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
Restore:
|
||||||
|
- Single file
|
||||||
|
- Database dump
|
||||||
|
- User folder sample
|
||||||
|
|
||||||
|
### Verify:
|
||||||
|
- File integrity
|
||||||
|
- Permissions
|
||||||
|
- Recovery speed
|
||||||
|
|
||||||
|
### Critical Principle
|
||||||
|
If restore testing is not performed, liability exposure increases substantially.
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Restore validation report
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4.4 Security Audit Review
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Weekly
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
30 minutes
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
#### Review:
|
||||||
|
- Admin accounts
|
||||||
|
- Group memberships
|
||||||
|
- External shares
|
||||||
|
- Public links
|
||||||
|
- Expired accounts
|
||||||
|
- MFA compliance
|
||||||
|
|
||||||
|
#### Validate:
|
||||||
|
- SSL certificate expiration dates
|
||||||
|
- Firewall rules
|
||||||
|
- SSH access
|
||||||
|
- Root login disabled
|
||||||
|
- Fail2Ban status (if implemented)
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Weekly security audit checklist
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4.5 Capacity and Performance Review
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Weekly
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
20–30 minutes
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
#### Analyze:
|
||||||
|
- Storage growth
|
||||||
|
- User growth
|
||||||
|
- Bandwidth usage
|
||||||
|
- CPU/RAM trends
|
||||||
|
- Database size growth
|
||||||
|
|
||||||
|
#### Evaluate:
|
||||||
|
- Need for droplet resize
|
||||||
|
- Need for archive policies
|
||||||
|
- Need for retention changes
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Capacity trend notes
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4.6 Documentation and Change Log
|
||||||
|
|
||||||
|
### Frequency
|
||||||
|
Weekly
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
15–20 minutes
|
||||||
|
|
||||||
|
### Objective
|
||||||
|
Maintain defensible operational records.
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
|
||||||
|
Document:
|
||||||
|
- Changes made
|
||||||
|
- Accounts added/removed
|
||||||
|
- Incidents
|
||||||
|
- Security events
|
||||||
|
- Backup issues
|
||||||
|
- Maintenance performed
|
||||||
|
|
||||||
|
### Important
|
||||||
|
Operational documentation is part of liability protection.
|
||||||
|
|
||||||
|
If a breach occurs, documented operational diligence matters significantly.
|
||||||
|
|
||||||
|
### Deliverable
|
||||||
|
- Weekly operational summary
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 5. Monthly Administrative Tasks
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5.1 Full Disaster Recovery Exercise
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
2–4 hours
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
Simulate:
|
||||||
|
- Server loss
|
||||||
|
- Container rebuild
|
||||||
|
- Restore from backup
|
||||||
|
- DNS validation
|
||||||
|
- SSL restoration
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5.2 User Access Certification
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
30–60 minutes
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
Review with client:
|
||||||
|
- Active users
|
||||||
|
- Admin privileges
|
||||||
|
- External sharing
|
||||||
|
- Terminated employees
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5.3 Security Policy Review
|
||||||
|
|
||||||
|
### Estimated Time
|
||||||
|
30 minutes
|
||||||
|
|
||||||
|
### Tasks
|
||||||
|
Review:
|
||||||
|
- MFA compliance
|
||||||
|
- Password standards
|
||||||
|
- Administrative access
|
||||||
|
- Training completion
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 6. Estimated Operational Effort
|
||||||
|
|
||||||
|
| Activity | Estimated Time |
|
||||||
|
|---|---|
|
||||||
|
| Daily Operations | 35–60 min/day |
|
||||||
|
| Weekly Maintenance | 2–4 hrs/week |
|
||||||
|
| Monthly DR/Security | 3–6 hrs/month |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 7. Recommended Retainer Guidance
|
||||||
|
|
||||||
|
For a pilot of this size:
|
||||||
|
|
||||||
|
| Service Level | Estimated Monthly Hours |
|
||||||
|
|---|---|
|
||||||
|
| Minimal Reactive Support | 8–10 hrs |
|
||||||
|
| Recommended Operational Support | 15–20 hrs |
|
||||||
|
| Security-Conscious Managed Support | 25–35 hrs |
|
||||||
|
|
||||||
|
Given the recent discussions around:
|
||||||
|
- liability
|
||||||
|
- data protection
|
||||||
|
- backup validation
|
||||||
|
- MFA enforcement
|
||||||
|
- user training
|
||||||
|
- documented diligence
|
||||||
|
|
||||||
|
…the “Recommended Operational Support” tier is likely the minimum responsible posture.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 8. Key Risk Areas to Monitor
|
||||||
|
|
||||||
|
The largest liability exposure areas are:
|
||||||
|
|
||||||
|
## Administrative Misconfiguration
|
||||||
|
- Incorrect sharing permissions
|
||||||
|
- Public links
|
||||||
|
- Excessive admin rights
|
||||||
|
|
||||||
|
## Backup Failure
|
||||||
|
- Silent backup corruption
|
||||||
|
- Unverified restores
|
||||||
|
|
||||||
|
## Credential Compromise
|
||||||
|
- Weak passwords
|
||||||
|
- MFA disabled
|
||||||
|
- Phishing
|
||||||
|
|
||||||
|
## Delayed Patching
|
||||||
|
- Unpatched Nextcloud vulnerabilities
|
||||||
|
- Docker/container CVEs
|
||||||
|
- Linux exploits
|
||||||
|
|
||||||
|
## User Behavior
|
||||||
|
- Unsafe uploads
|
||||||
|
- Credential reuse
|
||||||
|
- Local machine compromise
|
||||||
|
|
||||||
|
## Lack of Documentation
|
||||||
|
- No operational evidence
|
||||||
|
- No audit trail
|
||||||
|
- Undefined responsibilities
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 9. Strong Recommendations
|
||||||
|
|
||||||
|
## Require:
|
||||||
|
- MFA for all users
|
||||||
|
- Mandatory admin training
|
||||||
|
- Signed acceptable use/security acknowledgment
|
||||||
|
- Principle of least privilege
|
||||||
|
|
||||||
|
## Strongly Recommended:
|
||||||
|
- Centralized logging
|
||||||
|
- Automated monitoring alerts
|
||||||
|
- Offsite backups
|
||||||
|
- Written incident response plan
|
||||||
|
- Cyber liability / E&O insurance
|
||||||
|
|
||||||
|
## Avoid:
|
||||||
|
- Shared admin accounts
|
||||||
|
- Permanent public links
|
||||||
|
- Unrestricted upload folders
|
||||||
|
- Direct root SSH access
|
||||||
|
- Unmanaged personal devices for administrators
|
||||||
@@ -67,18 +67,18 @@ If your organization enforces security policies, you may be automatically logged
|
|||||||
|
|
||||||
If you ever receive an error such as “invalid credentials,” carefully re-enter your username and password. If the issue persists, use the password reset option or contact support.
|
If you ever receive an error such as “invalid credentials,” carefully re-enter your username and password. If the issue persists, use the password reset option or contact support.
|
||||||
|
|
||||||
1.6 Accessing Nextcloud from Other Devices (Overview)
|
### 1.6 Accessing Nextcloud from Other Devices (Overview)
|
||||||
|
|
||||||
While this guide focuses on web access, it is important to understand that Nextcloud is not limited to your browser.
|
While this guide focuses on web access, it is important to understand that Nextcloud is not limited to your browser.
|
||||||
|
|
||||||
You may also access your files through:
|
You may also access your files through:
|
||||||
|
|
||||||
A desktop sync application installed on your computer
|
- A desktop sync application installed on your computer
|
||||||
A mobile app on your phone or tablet
|
- A mobile app on your phone or tablet
|
||||||
|
|
||||||
These options allow you to work with files more seamlessly, but for initial setup and training, the web interface is the standard starting point.
|
These options allow you to work with files more seamlessly, but for initial setup and training, the web interface is the standard starting point.
|
||||||
|
|
||||||
1.7 Common First-Time Issues and How to Respond
|
### 1.7 Common First-Time Issues and How to Respond
|
||||||
|
|
||||||
It is normal to encounter minor issues during your first login. Below are common scenarios:
|
It is normal to encounter minor issues during your first login. Below are common scenarios:
|
||||||
|
|
||||||
@@ -93,7 +93,7 @@ These are part of your organization’s security setup—follow the instructions
|
|||||||
|
|
||||||
If you are unsure at any step, pause and reach out to your support contact rather than guessing.
|
If you are unsure at any step, pause and reach out to your support contact rather than guessing.
|
||||||
|
|
||||||
1.8 Key Takeaways
|
### 1.8 Key Takeaways
|
||||||
Nextcloud is your organization’s secure file and collaboration environment.
|
Nextcloud is your organization’s secure file and collaboration environment.
|
||||||
Always use the official URL provided to you.
|
Always use the official URL provided to you.
|
||||||
Your first login establishes your personal security settings.
|
Your first login establishes your personal security settings.
|
||||||
|
|||||||
Reference in New Issue
Block a user