Operations
This guide covers the day-two work for the AWS AMI stack: health, logs, deployments, backups, scaling, and security. The Docker, GCP, and Azure guides cover the equivalent commands for those paths.
Health checks
| Check | Command |
|---|---|
| Application health | curl -fsS https://your-host/up |
| Nginx | sudo systemctl status nginx |
| PHP-FPM | sudo systemctl status php8.3-fpm |
| Queue worker | sudo systemctl status hris-queue |
| Scheduler | systemctl list-timers hris-scheduler.timer |
| Backup timer | systemctl list-timers hris-backup.timer |
| Payroll engine | sudo -u www-data php /var/www/hris/artisan payroll:engine --check |
| Schema verification | sudo -u www-data php /var/www/hris/artisan db:ensure-schema --check |
| ALB target health | aws elbv2 describe-target-health --target-group-arn <arn> |
The ALB target group uses /up with a 200 matcher. Keep that route healthy; it is the fastest signal that PHP, the database, and the application can serve traffic.
Logs
# Application
sudo tail -f /var/www/hris/storage/logs/laravel.log
sudo tail -f /var/www/hris/storage/logs/laravel-$(date +%F).log
# Bootstrap
sudo tail -f /var/log/hris-bootstrap.log
# Nginx
sudo tail -f /var/log/nginx/hris-access.log
sudo tail -f /var/log/nginx/hris-error.log
# PHP-FPM
sudo journalctl -u php8.3-fpm -f
sudo tail -f /var/log/php-fpm/hris-error.log
# Queue and scheduler
sudo journalctl -u hris-queue -f
sudo journalctl -u hris-scheduler -n 100 --no-pager
Logs are on the instance. Add the CloudWatch agent or ship them to your observability platform if you need retention beyond the instance lifecycle.
Deployments
Golden image deployment
- Build a new AMI at the release tag.
- Set
app_refinterraform.tfvarsto the same tag. - Run
terraform apply. - Watch
/var/log/hris-bootstrap.logand/up. - Keep the previous AMI until the new instance is healthy.
In-place deployment
sudo /usr/local/bin/hris-deploy v1.1.0
The helper:
- Fetches the ref.
- Installs production Composer dependencies.
- Rebuilds Vite assets.
- Rebuilds the Go payroll binary.
- Runs migrations.
- Imports the statutory tables and 2026 wage orders.
- Warms config, event, and view caches.
- Restarts PHP-FPM, Nginx, and the queue worker.
For zero-downtime releases with an ALB, use an Auto Scaling group with instance refresh or a blue/green target group. The current stack is a single instance and will briefly restart services during an in-place deploy.
Database operations
Migrations
Schema changes ship as Laravel migrations in database/migrations; there is no SQL dump to import. A new database is created entirely by:
sudo -u www-data php /var/www/hris/artisan migrate --force
Always take a snapshot before a release that changes the schema or payroll logic.
What the current migrations add
- Baseline (
2026_10_01_*) — organisations, users and access, employees, attendance, compensation, payroll, contributions, and infrastructure, including the Filament tables. - Tenancy and infrastructure (
2026_11_02_*) — tenant columns, backfills,NOT NULLenforcement, and the sessions, cache, and queue tables. - Payroll compliance controls (
2026_11_04_*) — payroll revisions, audit events, the year-to-date ledger, the de minimis catalogue, and journal entries. - Agent conversations (
2026_11_06_*) — conversation and message tables for the AI assistant. - 2026 payroll controls (
2026_11_10_*) — statutory source status, wage-order and industry columns, payroll review/override columns, and nullable global minimum-wage rows. - Device credentials (
2026_11_12_*) —api_key,is_active, andlast_seen_atondevicesfor time-clock pushes.
php artisan migrate --force applies all of them; no manual SQL is required.
Upgrading an existing database
The migrations are safe on databases created before the SQL dumps were retired: they use Schema::hasTable and Schema::hasColumn guards, so they create only what is missing. Laravel records applied migrations in the migrations table, which it creates on the first run.
If a migration still fails because an object already exists outside the migration history:
- Back up the database.
- Inspect the failing migration and confirm its changes are already present.
- Record it as applied and continue:
sudo -u www-data php /var/www/hris/artisan tinker --execute 'DB::table("migrations")->insert(["migration" => "2026_10_01_000100_create_organization_tables", "batch" => ((int) DB::table("migrations")->max("batch")) + 1]);'
- Rerun
php artisan migrate --force.
Never mark a migration as applied unless its tables, columns, and indexes are actually present. Replace the example migration name with the one that failed.
Storage engines and character sets
Databases that started from the retired SQL dumps can still contain MyISAM tables and latin1 or utf8mb3 columns. MySQL rejects writes to a non-transactional table inside a transaction when GTID consistency is enforced (Laravel Cloud), MyISAM ignores rollbacks, and the legacy character sets cannot store every UTF-8 character or compare cleanly against the utf8mb4 tables Laravel creates. Two migrations repair this when php artisan migrate --force runs:
2026_11_19_235252_convert_non_transactional_tables_to_innodbrebuilds every table with InnoDB.2026_11_20_000111_convert_tables_to_utf8mb4converts every table and its character columns to the connection's configured character set and collation (utf8mb4/utf8mb4_unicode_ciby default).
Both statements copy each table, so take a snapshot first and expect a metadata lock and up to twice the table size in free disk space. On a large database, convert ahead of the deploy from a console so the migrations only have to verify:
# List the tables that are not InnoDB or not utf8mb4; exits non-zero when any remain.
sudo -u www-data php /var/www/hris/artisan db:ensure-schema --check
# Convert them in place.
sudo -u www-data php /var/www/hris/artisan db:ensure-schema
Installation configuration
On a fresh database, configure the installation once:
sudo -u www-data php /var/www/hris/artisan openhr:install \
--sector=Government \
--country=PH \
--no-interaction
Use --sector=Private for a private-sector installation. The command marks every existing company
and every company created later; it seeds the sector's employment, leave, deduction, loan and
compensation catalogues and imports the selected country's statutory schedule. If the database
already contains processed payrolls, changing the country requires --force and should be treated
as a migration project, not a routine deploy.
Leave types and compensations are system-wide catalogues: a shared row has a null company_id, and
a company row is either an override or a type only that company uses. The optional sector column
limits system rows to Government or Private; an empty value applies to both sectors.
Reference data after a migration
sudo -u www-data php /var/www/hris/artisan db:seed --class='Database\Seeders\ReferenceDataSeeder' --force
sudo -u www-data php /var/www/hris/artisan payroll:statutory:import --country=PH
sudo -u www-data php /var/www/hris/artisan payroll:wage-orders:import
sudo -u www-data php /var/www/hris/artisan payroll:holidays:import
All four commands are idempotent. ReferenceDataSeeder seeds the system-wide leave type and
compensation catalogues for the configured sector without creating a demo tenant. Leave out the
wage-order import on a Government installation: government compensation follows the DBM salary
schedule, and the bootstraps skip the import when OPENHR_SECTOR=Government.
payroll:holidays:import is company-scoped and fails until the first company exists, so run it
again after admin:create on a fresh installation.
Statutory table refresh
sudo -u www-data php /var/www/hris/artisan payroll:statutory:import --country=PH
sudo -u www-data php /var/www/hris/artisan payroll:statutory:import --country=PH --fresh
--fresh deletes the selected country's statutory rows before importing. Use it only when you
intend to replace the whole schedule. For a generic country:
sudo -u www-data php /var/www/hris/artisan payroll:statutory:import --country=XX --file=/path/components.json
Deployments with processed payrolls must keep the old effective-dated rows; a component import adds a new effectivity rather than rewriting history.
Backups and restore
RDS automated backups are enabled. For a manual snapshot:
aws rds create-db-snapshot \
--db-instance-identifier hris-production-db \
--db-snapshot-identifier hris-production-manual-$(date +%Y%m%d%H%M)
Restore by creating a new RDS instance from the snapshot, then point DB_HOST and credentials at it. Update the SSM parameter and rerun the bootstrap or edit /var/www/hris/.env and warm caches.
Restore uploaded files
aws s3 sync s3://<bucket>/storage/public /var/www/hris/storage/app/public
sudo chown -R www-data:www-data /var/www/hris/storage/app/public
Queues and schedulers
Payroll runs above PAYROLL_ENGINE_INLINE_THRESHOLD are queued. Keep at least one worker running:
sudo systemctl restart hris-queue
sudo journalctl -u hris-queue -n 200 --no-pager
The scheduler runs every minute through hris-scheduler.timer:
sudo systemctl start hris-scheduler.timer
systemctl list-timers hris-scheduler.timer
If a deploy changes job code, restart the worker so it picks up the new code.
Scaling
The current stack is a single EC2 instance. Before adding more instances:
- Move sessions and cache to a shared store (the default is already the database; Redis is a better choice at scale).
- Move media to S3 (
MEDIA_DISK=s3) or a shared EFS mount. - Run the queue worker and scheduler on exactly one instance, or move them to a separate worker tier.
- Put the instances behind the ALB and use health checks.
- Use RDS Multi-AZ for the database.
- Consider a separate build/deploy pipeline instead of in-place deploys.
Vertical scaling is the simplest first step: increase instance_type and db_instance_class and apply.
Security maintenance
- Patch the AMI monthly or when Ubuntu publishes security updates. Rebuild the image and roll it out.
- Keep
APP_DEBUG=falsein production. - Keep the RDS instance private.
- Use Session Manager instead of opening SSH. If SSH is required, restrict
ssh_allowed_cidrs. - Rotate the database password and
APP_KEYdeliberately. RotatingAPP_KEYcan invalidate encrypted data and sessions. - Rotate secrets in SSM, then rerun the bootstrap or update
/var/www/hris/.envand warm caches. - Keep the Terraform state encrypted and access-controlled.
- Review the open-source release checklist before sharing infrastructure code.
Incident checklist
- Check
/upand the ALB target health. - Read the latest Laravel and Nginx errors.
- Confirm the database is reachable and has free storage.
- Confirm the queue worker and scheduler are running.
- Confirm disk space:
df -h. - Confirm the payroll binary exists and is executable.
- If a release caused the incident, redeploy the previous ref or AMI.
- Restore from the latest RDS snapshot only after preserving the failed state for investigation.
Teardown
For a disposable environment, set:
deletion_protection = false
skip_final_snapshot = true
s3_force_destroy = true
Then run terraform destroy. Deregister unused AMIs and delete old snapshots separately.